it is more efficient to read at least 32bit at once. did you try to process 4 or 8 neighborhood pixels at once?