One view derives from the perspective of the raster map as a rectangular matrix of numbers that can be subjected to various mathematical operations. This viewpoint is similar to image processing applied to raster maps. An alternative view is fully articulated in C. Dana Tomlin's book, GIS and Cartographic Modeling. Tomlin doesn't diminish the mathematical perspective, but structures it so spatial processes and attributes are highlighted. Tomlin neighborhood operators include the ability to define irregular neighborhoods and neighborhoods with holes as well as compute values such as mode, least-common value, number of different values in the neighborhood, etc.
Connectivity also is an important neighborhood property.
In this operation, it's possible to determine the degree to which a single value in a map is adjacent to other cells of similar value. There's a lot of overlap among these operations. For example, a mathematical filter can be created that defines an irregular neighborhood. ArcView Spatial Analyst 2 includes a visual model-building tool kit. They also allow application of standard image processing filters as well as operators that identify diversity.