Locally orderless networks

Venue: NLDL_org_2025_Conference
Published: 2024-09-08 14:20:45

Abstract: We present Locally Orderless Networks (LON) and the theoretical foundation that links them to Convolutional Neural Networks (CNN), Scale-space histograms, and measurement theory. The key elements are a regular sampling of the bias and the derivative of the activation function. We compare LON, CNN, and Scale-space histograms on prototypical single-layer networks. We show how LON and CNN can emulate each other and how LON expands the set of functions computable to non-linear functions such as squaring. We demonstrate simple networks that illustrate the improved performance of LON over CNN on simple tasks for estimating the gradient magnitude squared, for regressing shape area and perimeter lengths, and for explainability of individual pixels' influence on the result.

Main paper: 
 introduction: 
 We introduce the locally orderless network (LON) which locally transforms the input signal into a set of local histograms. We propose a small modification of a standard convolutional neural network such that the internal representations becomes local histograms, as they have been studied in the classical scale-space literature [1] . This novel representation includes learnable operators which expands the representation space to a wide group of non-linear functions.In [1] , the authors presents the Locally Orderless Image (LOI) which identifies the 3 essential scale parameters of any gray-valued image: the resolution of the image, the size of the region of analysis, and intensity resolution. The authors show how these concepts are unified into a single scale-space paradigm, and at its heart resides local histograms which are the smooth and well-posed. With the strong foundation of classic scale space and measure theory [2, 3] , in this article we illustrate how basic theoretical perspectives can be used to analyze these types of networks including classic concepts of invariance and density estimates. We focus on understanding the fundamental capacities of the LON and comparing it to Convolutional Neural Networks (CNN) [4] . Thus we describe the LON from a theoretical perspective and compare it with a similar CNN on simple binary examples to illustrate some of their properties.To the best of our knowledge, there is little prior 048 work that is closely related to the LON. Histograms 049 have been used in networks previously, where [5] 050 introduced the histogram layer for more compact 051 representation and realized that this is induced by 052 the activation function. In [6] a global version was in-053 troduced, and [7] showed how such a representation 054 can be used to estimate quantiles and subsequently 055 distance. Density estimation deep learning network 056 components exist, but most of that relates to den-057 sity for loss functions such as cross-entropy. Pooling 058 operation which can be interpreted as a histogram 059 operation, and the learning of pooling operations 060 was suggested by [8] where the pooling operation 061 was learned during training. As an alternative one 062 could consider Bayesian Neural Networks [9] that 063 work with parameter uncertainty or distributions. In 064 contrast, LON works with data distribution. There 065 are however some works before the breakthrough of 066 neural networks. In [10, 11] , local histograms were 067 used for texture classification with success. Further-068 more, many of the classical image descriptors such 069 as histogram of Gaussians (HoG), Daisy [12] , and 070 SIFT [13] are all based on histograms of features in 071 some way. noise, the circumference and area of S is,EQUATION) 079 written as h(x, b) = (W * f (b -(I * K)))(x). When 080 W is a smoothing kernel,EQUATIONLON ({I k }) = A vec({h ijk }) (3) 095 where Θ = {K j , b ij , σ ij , W j , A : i = 1 . . . N, j = 096 1 . . . M } is the set of parameters, A ∈ R M N(v) = 108 v -∞ f ij (w) dw, 109 CNN({I k }) (4) 110 = A vec({(W j * g ij (b ij -(I * K j )))(x k )}) .EQUATIONEQUATIONEQUATIONEQUATION)127where K is a smoothing kernel, and A are implied 128 to be a 1-vector with |Ω| elements. EQUATIONA local version is obtained by convolution with a smoothing kernel W ,EQUATIONsince the above is linear in h then it can be written on the form (3). Further, any linear combinations of transformationsEQUATION)is also linear in h and thus can also be written on the form (3).As an example, consider derivative kernels, K k (x) where x = (x 1 , x 2 , . . . , x d ), and such that I k = I * K k is a smoothed estimate of the directional derivative in the x k -direction. With ξ(v) = v 2 , the LON can approximate the gradient magnitude squared as,EQUATIONNote that for linear functions ξ and Gaussian kernels W and K or its derivatives with standard deviation γ and σ, then convolution semi-group property of Gaussian kernels implies that the two kernels can be replaced with a single Gaussian kernel of standard deviation γ 2 + σ 2 with an appropriate sum of their derivative orders. This does not hold when ξ is non-linear, but our experience is that the resulting scale of grad 2 is close to γ 2 + σ 2 . 
experiments: 
 In this section, we will compare LON (3) with CNN (5). To focus on the inner parts of the networks, we set W j = δ, where δ is the Dirac delta function, and we investigate their ability to estimate the gradient magnitude squared, to estimate and classify circumferences and areas of objects, and we perform a sensitivity analysis.The gradient magnitude squared is a rotational invariant indicator of the apparent edge of object parts. Using spatial coordinates x = [x, y], a direct implementation is given by |I(x)| For the regression task, we see wrt. the perimeter, the LON outperforms CNN in the 2-bin case in terms of the number of parameters used, however, for the 8-bin case it seems that the LON overfits.Wrt. area, LON with 2 bins has trouble converging, while CNN outperforms both LON.For the classification task, we divided the objects into small, medium, and large wrt. either the area or perimeter length, and asked the network to correctly that LON with 2 bins is better at classifying objects 267 in terms of their perimeter, while CNN is better at 268 classifying objects wrt. area. In this experiment, we consider the classification 285 task on random, noise-free shapes, shown in Fig- 286 ure 3(c). The resulting sensitivity image is shown 287 for various combinations of networks, channels, and 288 the essential number of bins in Figure 6 . It shows 289 that although all the models can make correct pre-290 dictions with very high accuracy, as demonstrated 291 in Figure 5 CNN on the other hand is inferring the perimeter 296 from both the foreground and the background. explainability. Future work will explore combining LONs with existing networks, particularly for segmentation tasks. 
 
 Appendix:  


Decision: The reviewers agree that the paper is well-motivated and has a good theoretical foundation. Reviewers, however, also differ in their opinions, with two positive reviewers, one neutral, and one that recommends a rejection. The strong aspects of the paper are its theoretical foundation, that it is based on a clear, intuitive, and simple idea, and that it is well described. The highlighted weaknesses are limitations of the experimental validation, limited scope of the method, limited novelty, and insufficient discussion of computational complexity, scalability, and generalizability. 

The authors have given a thorough rebuttal that addresses all comments from the reviewers and rewritten the paper accordingly. So, with the primarily positive reviews, I recommend accepting the paper as a poster.

Reviews: review: title: title: Locally Orderless Networks
paper_summary: paper_summary: The paper proposes an architecture called "Locally Orderless Networks" (LONs), which is analogous to CNNs with biases but uses Gaussian functions as activations. The authors note that sigmoid functions, which are common activations in CNNs, can be viewed as integrals of Gaussian functions. Therefore, while a CNN layer performs soft thresholding of the convolved input, a LON layer implements a "soft indicator" of the convolved input. According to the authors, this property makes LONs a better choice for tasks, which involve estimating lengths of curves of the same intensity in images (such as perimeter estimation), while CNNs are more appropriate for area estimation tasks. The paper provides experiments to support this intuition, where LONs marginally outperform CNNs for perimeter estimation. Moreover, the authors suggest that LONs result in more intuitive saliency maps in comparison with CNNs, and thus provide better explainability.
main_review: main_review: Since I already recommended acceptance in the initial review, there is no change in my assessment post rebuttal.
strength_weakness: strength_weakness: **W1**: The novelty of the paper is marginal, given that the LON architecture amounts to simply using a different activation function in a CNN.

**W2**:  The writing could be significantly improved, especially in the introduction and conclusion.

**Questions/suggestions**:
* **Q1**: In the proposed architecture, the outer layer has the number of parameters proportional to the image size $|\Omega|$. I.e., every spatial location of the convolved image is multiplied by an independent set of weights. However, since the goal of the main experiments is just to estimate the perimeter or area, it should not be important where exactly the object appears in the picture. Therefore, shouldn't it be sufficient to use the same parameters for all the spatial locations (i.e., use a variant of a convolution in the outer layer)? Why do the authors choose to train seemingly much more parameters than needed for these tasks? 
* **Q2**: Which exact LON is displayed in Figure 1? What are the kernels parameters?
* Figures 4 and 5 would benefit from bigger legend and font sizes.
* **Q3**: Figure 6 only shows correctly classified samples. How would missclassified samples look? 
* Classes «0,1,2» should be defined in Figure 6 (I assume they correspond to area/parameter classes from the smallest to the largest?).
* **Q4**: How do the authors interpret the behaviour of LON with 8 bins and 2 kernels in Figure 6? How does it relate to the mentioned "overfitting" in Figure 4? Is it related to the much larger parameters count than needed (see **Q1**)?

**Writing problems/typos**:
* The dimensions of $A$ in line 089 should be transposed. 
* There are many unclear/undefined wording choices in the introduction. E.g.: "function width" (I assume it is $\sigma$, but the word choice in confusing in NNs context), "local histogram", "scale space", "activity/activation" (are these used interchangeably?), "sigmoid derivatives» (is this the same as Gaussian functions?).
* In my view, the locally orderless images framework and its relationship to the proposed architecture is not explored and explained enough in the text. However, I understand that it is partially due to the limited space. 
* There are some language mistakes. E.g., "network components exists" (line 049), "LON expresses more naturally to some local operators than CNN" (line 123), etc. 
* The conclusion is especially poorly written and includes sentences that are incorrect and misleading. E.g., "... layer, called Locally Orderless Networks (LONs), allows CNNs and LONs to model each other and compute non-linear functions, like squaring".
title: title: Locally Orderless Networks - Strengths in theoretical foundations and weaknesses in experimental scope and generalizability
paper_summary: paper_summary: The paper "Locally Orderless Networks" (LONs) introduces a novel neural network architecture that integrates scale-space theory and measure theory, enhancing the computational capabilities of neural networks by employing locally orderless histograms. It establishes LONs as a generalization of the Locally Orderless Image framework, allowing for the computation of non-linear functions like squaring. The research demonstrates that LONs outperform Convolutional Neural Networks (CNNs) in tasks related to edge detection, gradient magnitude estimation, and shape analysis, showcasing superior performance in object classification based on perimeter and improved explainability through saliency maps. Overall, LONs offer a theoretically sound and effective alternative to CNNs, particularly for boundary-focused tasks, with potential for further integration in future work.
main_review: main_review: The authors have satisfactorily addressed all of my comments and have made appropriate changes to strengthen the critical aspects of the paper. Therefore, my recommendation is to accept the paper for publication.
strength_weakness: strength_weakness: 1. The experiments primarily focus on gradient magnitude estimation and shape analysis, lacking a broader range of tasks and datasets. Expanding the experimental scope would strengthen claims about the versatility and generalizability of LONs.

2. The paper compares LONs with CNNs and does not include a broader analysis against other state-of-the-art architectures.

3. While the experiments indicate that LONs may exhibit overfitting, the paper does not adequately address methods to mitigate this issue. Incorporating techniques like regularization or cross-validation would enhance the reliability of the findings.

4. The paper lacks a thorough discussion on the scalability and computational efficiency of LONs, particularly regarding the added parameters from locally orderless histograms.

5. Although the theoretical foundations are solid, the paper does not explore the limitations and assumptions of LONs in depth. Understanding the conditions under which LONs might fail or produce suboptimal results would enhance credibility.

6. Certain terminology and concepts, such as scale-space and measure theory, should be better explained.

7. The font size of Figures 1, 4, and 5 is very small and cannot be read.
title: title: Locally Orderless Networks as a generalization of  Convolutional Neural Networks
paper_summary: paper_summary: This paper introduces a novel neural network layer called Locally Orderless Networks (LON) and explores its theoretical connections with Convolutional Neural Networks (CNN) and scale-space histograms. The core of LON lies in the regular sampling of the bias and the derivative of the activation function to create local histograms. By comparing LON, CNN, and scale-space histograms on prototypical single-layer networks, the paper demonstrates LON's advantages in specific tasks such as gradient magnitude squared estimation, shape area, and perimeter length regression, as well as explainability of individual pixel influence on the results.
main_review: main_review: It is difficult for me to assess whether the strengths and weaknesses of the model presented in this paper are critical.
strength_weakness: strength_weakness: The weaknesses of the paper encompass the increased complexity and computational demands of LON due to its higher number of parameters compared to CNNs, concerns about its generalization ability beyond the specific tasks and datasets explored, the potential for overfitting highlighted by certain experiments, and the need for broader experimental validation to ascertain LON's versatility and practicality in diverse applications.
title: title: Locally Orderless Networks - Interesting approach with narrow / limited results
paper_summary: paper_summary: The article introduces Locally Orderless Networks; operators with roots in Locally Orderless Image frameworks. The work draws on kernel methods via general Parzen-Rosenblatt windows and local histograms, and connect these to the general convolutional operator from CNNs. The link is well grounded in theory, and the authors reference earlier work with local feature descriptors (SIFT, HoG).  

Compared to CNNs, LON emphasizes soft isophotes (boundaries of regions of constant value), whereas CNNs perform thresholding. This can potentially make LON effective in certain tasks where CNNs might theoretically be less well suited, such as shape regression. The authors also investigate LONs for gradient based saliency maps.
main_review: main_review: We appreciate that the authors are looking to establish the feasibility of a more theoretical approach. We believe the idea could have merit, as discussed in our review, however, the experiments designed around the current approach does not clearly show or delineate significant benefits of the proposed approach.

In summary, we believe the rebuttal fails to sufficiently establish the value of the proposed LON network layer in a modern modeling framework. It is still not sufficiently clear to the reader how LONs provide benefits compared to a standard convolutional layer, and the authors comments are not sufficiently addressing this limitation. For this reason, we will maintain our recommendation to not accept the work as part of the conference.

We wish the authors luck in their future work towards locally orderless network layers.
strength_weakness: strength_weakness: (W1) The experiments are limited in scope, and its general applicability to more complex or diverse tasks remains mostly unexplored. It is not clear from the experiments that LONs offer significant benefits in general modelling settings. This is by far the most pressing weakness of the work, as it provides little incentive for direct applications.

(W2) It seems that LON requires more parameters compared to CNNs, especially as the number of bins increases and particularly in the area regressor experiment. While this is somewhat discussed, the current results seem to indicate issues for LONs with higher number of bins. As a result, the reader is left with more questions regarding when exactly this operator could serve as viable modelling tool.

(W3) While the saliency maps provided by LONs are more focused, the definition of saliency with pure gradients is not considered robust. In this reviewers opinion, it somewhat demonstrates that LONs seems well suited to shape regression. However, pure gradients as a general saliency method has been demonstrated to be unreliable in more general settings [(Adebayo et al. 2018)](https://arxiv.org/abs/1810.03292).