Monotonic Differentiable Sorting Networks

Differentiable sorting algorithms allow training with sorting and ranking supervision , where only the ordering or ranking of samples is known . Various methods have been proposed to address this challenge , ranging from optimal transport-based differentiable Sinkhorn sorting algorithms to making classic sorting networks differentiable . One problem of current differentiable sorting methods is that they are non-monotonic . To address this issue , we propose a novel relaxation of conditional swap operations that guarantees monotonicity in differentiable sorting networks . We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic . Monotonicity ensures that the gradients always have the correct sign , which is an advantage in gradient-based optimization . We demonstrate that monotonic differentiable sorting networks improve upon previous differentiable sorting methods . 1 INTRODUCTION . Recently , the idea of end-to-end training of neural networks with ordering supervision by a continuous relaxation of the sorting function has been presented by Grover et al . [ 1 ] . The idea of ordering supervision is that the ground truth order of some samples is known while their absolute values remain unsupervised . This is done by integrating a sorting algorithm in the neural architecture . As for training with a sorting algorithm in the architecture , the error needs to be propagated in a meaningful way back to the neural network , it is necessary to use a differentiable sorting function . Several such differentiable sorting functions have been introduced , e.g. , by Grover et al . [ 1 ] , Cuturi et al . [ 2 ] , Blondel et al . [ 3 ] , and Petersen et al . [ 4 ] . In this work , we focus on analyzing differentiable sorting functions [ 1 ] – [ 4 ] and demonstrate how monotonicity improves differentiable sorting networks [ 4 ] . Sorting networks are a family of sorting algorithms that consist of two basic components : so called “ wires ” ( or “ lanes ” ) carrying values , and conditional swap operations that connect pairs of wires [ 5 ] . An example of such a sorting network is shown in the center of Figure 1 . The conditional swap operations swap the values carried by these wires if they are not in the desired order . They allow for fast hardware-implementation , e.g. , in ASICs , as well as on highly parallelized general-purpose hardware like GPUs . Differentiable sorting networks [ 4 ] continuously relax the conditional swap operations by relaxing their step function to a logistic sigmoid function . One problem that arises in this context is that using a logistic sigmoid function does not preserve monotonicity of the relaxed sorting operation , which can cause gradients with the wrong sign . In this work , we present a family of sigmoid functions that preserve monotonicity of differentiable sorting networks . These include the cumulative density function ( CDF ) of the Cauchy distribution , as well as a function that minimizes the error-bound and thus induces the smallest possible approximation error . For all sigmoid functions , we prove and visualize the respective properties and validate their advantages empirically . In fact , by making the sorting function monotonic , it also becomes quasiconvex , which has been shown to produce favorable convergence rates [ 6 ] . In Figure 2 , we demonstrate monotonicity for different choices of sigmoid functions . As can be seen in Figure 4 , existing differentiable sorting operators are either non-monotonic or have an unbounded error . Following recent work [ 1 ] , [ 2 ] , [ 4 ] , we benchmark our continuous relaxations by predicting values displayed on four-digit MNIST images [ 7 ] supervised only by their ground truth order . The evaluation shows that our method outperforms existing relaxations of the sorting function on the four-digit MNIST ordering task as well as the SVHN ranking task . Contributions . In this work , we show that sigmoid functions with specific characteristics produce monotonic and error-bounded differentiable sorting networks . We provide theoretical guarantees for these functions and also give the monotonic function that minimizes the approximation error . We empirically demonstrate that the proposed functions improve performance . 2 RELATED WORK . Recently , differentiable approximations of the sorting function for weak supervision were introduced by Grover et al . [ 1 ] , Cuturi et al . [ 2 ] , Blondel et al . [ 3 ] , and Petersen et al . [ 4 ] . In 2019 , Grover et al . [ 1 ] proposed NeuralSort , a continuous relaxation of the argsort operator . A ( hard ) permutation matrix is a square matrix with entries 0 and 1 such that every row and every column sums up to 1 , which defines the permutation necessary to sort a sequence . Grover et al . relax hard permutation matrices by approximating them as unimodal row-stochastic matrices . This relaxation allows for gradient-based stochastic optimization . On various tasks , including sorting four-digit MNIST numbers , they benchmark their relaxation against the Sinkhorn and Gumbel-Sinkhorn approaches proposed by Mena et al . [ 8 ] . Cuturi et al . [ 2 ] follow this idea and approach differentiable sorting by smoothed ranking and sorting operators using optimal transport . As the optimal transport problem alone is costly , they regularize it and solve it using the Sinkhorn algorithm [ 9 ] . By relaxing the permutation matrix , which sorts a sequence of scalars , they also train a scalar predictor of values displayed by four-digit numbers while supervising their relative order only . Blondel et al . [ 3 ] cast the problem of sorting and ranking as a linear program over a permutahedron . To smooth the resulting discontinuous function and provide useful derivatives , they introduce a strongly convex regularization . They evaluate the proposed approach in the context of Top-k classification and label ranking accuracy via a soft Spearman ’ s rank correlation coefficient . Recently , Petersen et al . [ 4 ] proposed differentiable sorting networks , a differentiable sorting operator based on sorting networks with differentiably relaxed conditional swap operations . Differentiable sorting networks achieved a new state-of-the-art on both the four-digit MNIST sorting benchmark and the SVHN sorting benchmark . Applications and Broader Impact . In the domain of recommender systems , Lee et al . [ 10 ] propose differentiable ranking metrics , and Swezey et al . [ 11 ] propose PiRank , a learning-to-rank method using differentiable sorting . Other works explore differentiable sorting-based top-k for applications such as differentiable image patch selection [ 12 ] , differentiable k-nearest-neighbor [ 1 ] , [ 13 ] , top-k attention for machine translation [ 13 ] , and differentiable beam search methods [ 13 ] , [ 14 ] . 3 BACKGROUND : SORTING NETWORKS . Sorting networks have a long tradition in computer science since the 1950s [ 5 ] . They are highly parallel data-oblivious sorting algorithms . They are based on so-called conditional pairwise swap operators that map two inputs to two outputs and ensure that these outputs are in a specific order . This is achieved by simply passing through the inputs if they are already in the desired order and swapping them otherwise . The order of executing conditional swaps is independent of the input values , which makes them data-oblivious . Conditional swap operators can be implemented using only min and max . That is , if the inputs are a and b and the outputs a′ and b′ , a swap operator ensures a′ ≤ b′ and can easily be formalized as a′ = min ( a , b ) and b′ = max ( a , b ) . Examples of sorting nets are the odd-even network [ 15 ] , which alternatingly swaps odd and even wires with their successors , and the bitonic network [ 16 ] , which repeatedly merges and sorts bitonic sequences . While to odd-even network requires n layers , the bitonic networks uses the divide-and-conquer principle to sort within only ( log2 n ) ( 1 + log2 n ) /2 layers . Note that , while they are similar in name , sorting networks are not neural networks that sort . 3.1 DIFFERENTIABLE SORTING NETWORKS . In the following , we recapitulate the core concepts of differentiable sorting networks [ 4 ] . An example of an odd-even sorting network is shown in the center of Figure 1 . Here , odd and even neighbors are conditionally swapped until the entire sequence is sorted . Each conditional swap operation can be defined in terms of min and max as detailed in Section 3 . These operators can be relaxed to differentiable min and max . Note that we denote the differentiable relaxations in italic font and their hard counterparts in roman font . Note that the differentiable relaxations min and max are different from the commonly used softmin and softmax , which are relaxations of argmin and argmax [ 17 ] . One example for such a relaxation of minσ and maxσ is the logistic relaxation minσ ( a , b ) = a · σ ( b− a ) + b · σ ( a− b ) and maxσ ( a , b ) = a · σ ( a− b ) + b · σ ( b− a ) ( 1 ) where σ is the logistic sigmoid function with inverse temperature β > 0 : σ : x 7→ 1 1 + e−βx . ( 2 ) Any layer of a sorting network can also be represented as a relaxed and doubly-stochastic permutation matrix . Multiplying these ( layer-wise ) permutation matrices yields a ( relaxed ) total permutation matrix P. Multiplying P with an input x yields the differentiably sorted vector x̂ = Px , which is also the output of the differentiable sorting network . Whether it is necessary to compute P , or whether x̂ suffices , depends on the specific application . For example , for a cross-entropy ranking / sorting loss as used in the experiments in Section 6 , P can be used to compute the cross-entropy to a ground truth permutation matrix Q . In the next section , we build on these concepts to introduce monotonic differentiable sorting networks , i.e. , all differentiably sorted outputs x̂ are non-decreasingly monotonic in all inputs x . 4 MONOTONIC DIFFERENTIABLE SORTING NETWORKS . In this section , we start by introducing definitions and building theorems upon them . In Section 4.2 , we use these definitions and properties to discuss different relaxations of sorting networks . 4.1 THEORY . We start by defining sigmoid functions and will then use them to define continuous conditional swaps . Definition 1 ( Sigmoid Function ) . We define a ( unipolar ) sigmoid ( i.e. , s-shaped ) function as a continuous monotonically non-decreasing odd-symmetric ( around 12 ) function f with f : R→ [ 0 , 1 ] with lim x→−∞ f ( x ) = 0 and lim x→∞ f ( x ) = 1 . Definition 2 ( Continuous Conditional Swaps ) . Following [ 4 ] , we define a continuous conditional swap in terms of a sigmoid function f : R→ [ 0 , 1 ] as minf ( a , b ) : = a · f ( b− a ) + b · f ( a− b ) max f ( a , b ) = a · f ( a− b ) + b · f ( b− a ) ( 3 ) argminf ( a , b ) = ( f ( b− a ) , f ( a− b ) ) argmax f ( a , b ) = ( f ( a− b ) , f ( b− a ) ) . ( 4 ) We require a continuous odd-symmetric sigmoid function to preserve most of the properties of min and max while also making argmin and argmax continuous as shown in Supplementary Material B . In the following , we establish doubly-stochasticity and differentiability of P , which are important properties for differentiable sorting and ranking operators . Theorem 3 ( Doubly-Stochasticity and Differentiability of P ) . ( i ) The relaxed permutation matrix P , produced by a differentiable sorting network , is doubly-stochastic . ( ii ) P has the same differentiability as f , e.g. , if f is continuously differentiable in the input , P will be continuously differentiable in the input to the sorting network . If f is differentiable almost everywhere ( a.e . ) , P will be diff . a.e . Proof . ( i ) For each conditional swap between two elements i , j , the relaxed permutation matrix is 1 at the diagonal except for rows i and j : at points i , i and j , j the value is v ∈ [ 0 , 1 ] , at points i , j and j , i the value is 1− v and all other entries are 0 . This is doubly-stochastic as all rows and columns add up to 1 by construction . As the product of doubly-stochastic matrices is doubly-stochastic , the relaxed permutation matrix P , produced by a differentiable sorting network , is doubly-stochastic . ( ii ) The composition of differentiable functions is differentiable and the addition and multiplication of differentiable functions is also differentiable . Thus , a sorting network is differentiable if the employed sigmoid function is differentiable . “ Differentiable ” may be replaced with any other form of differentiability , such as “ differentiable a.e. ” Now that we have established the ingredients to differentiable sorting networks , we can focus on the monotonicity of differentiable sorting networks . Definition 4 ( Monotonic Continuous Conditional Swaps ) . We say f produces monotonic conditional swaps if minf ( x , 0 ) is non-decreasingly monotonic in x , i.e. , min ′f ( x , 0 ) ≥ 0 for all x . It is sufficient to define it w.l.o.g . in terms of minf ( x , 0 ) due to its commutativity , stability , and odd-symmetry of the operators ( cf . SM ) . Theorem 5 ( Monotonicity of Continuous Conditional Swaps ) . A continuous conditional swap ( in terms of a differentiable sigmoid function f ) being non-decreasingly monotonic in all arguments and outputs requires that the derivative of f decays no faster than 1/x2 , i.e. , f ′ ( x ) ∈ Ω ( 1 x2 ) . ( 5 ) Proof . We show that Equation 5 is a necessary criterion for monotonicity of the conditional swap . Because f is a continuous sigmoid function with f : R→ [ 0 , 1 ] , minf ( x , 0 ) = f ( −x ) · x > 0 for some x > 0 . Thus , montononicity of minf ( x , 0 ) implies lim supx→∞minf ( x , 0 ) > 0 ( otherwise the value would decrease again from a value > 0 . ) Thus , lim x→∞ minf ( x , 0 ) = lim x→∞ f ( −x ) · x = lim x→∞ f ( −x ) 1/x ( L ’ Hôpital ’ s rule ) = lim x→∞ −f ′ ( −x ) −1/x2 ( 6 ) = lim x→∞ f ′ ( −x ) 1/x2 = lim x→∞ f ′ ( x ) 1/x2 = lim sup x→∞ f ′ ( x ) 1/x2 > 0⇐⇒ f ′ ( x ) ∈ Ω ( 1 x2 ) . ( 7 ) assuming limx→∞ f ′ ( x ) 1/x2 exists . Otherwise , it can be proven analogously via a proof by contradiction . Corollary 6 ( Monotonic Sorting Networks ) . If the individual conditional swaps of a sorting network are monotonic , the sorting network is also monotonic . Proof . If single layers g , h are non-decreasingly monotonic in all arguments and outputs , their composition h ◦ g is also non-decreasingly monotonic in all arguments and outputs . Thus , a network of arbitrarily many layers is non-decreasingly monotonic . Above , we formalized the property of monotonicity . Another important aspect is whether the error of the differentiable sorting network is bounded . It is very desirable to have a bounded error because without bounded errors the result of the differentiable sorting network diverges from the result of the hard sorting function . Minimizing this error is desirable . Definition 7 ( Error-Bounded Continuous Conditional Swaps ) . A continuous conditional swap has a bounded error if and only if supxminf ( x , 0 ) = c is finite . The continuous conditional swap is therefore said to have an error bounded by c. It is sufficient to define it w.l.o.g . in terms of minf ( x , 0 ) due to its commutativity , stability , and odd-symmetry of the operators ( cf . SM ) . In general , for better comparability between functions , we assume a Lipschitz continuous function f with Lipschitz constant 1 . Theorem 8 ( Error-Bounds of Continuous Conditional Swaps ) . ( i ) A differentiable continuous conditional swap has a bounded error if f ′ ( x ) ∈ O ( 1 x2 ) . ( 8 ) ( ii ) If it is additionally monotonic , the error bound can be found as limx→∞minf ( x , 0 ) and additionally the error is bound only if Equation 8 holds . Proof . ( i ) W.l.o.g . we consider x > 0 . Let g ( z ) : = f ( −1/z ) , g ( 0 ) = 0 . Thus , g′ ( z ) = 1/z2 · f ′ ( −1/z ) ≤ c according to Equation 8 . Thus , g ( z ) = g ( 0 ) + ∫ z 0 g′ ( t ) dt ≤ c · z . Therefore , f ( −1/z ) ≤ c · z =⇒ 1/z · f ( −1/z ) ≤ c and with x = 1/z =⇒ x · f ( −x ) = minf ( x , 0 ) ≤ c. ( ii ) Let minf ( x , 0 ) be monotonic and bound by minf ( x , 0 ) ≤ c. For x > 0 and h ( x ) : = minf ( x , 0 ) , h′ ( x ) = −x · f ′ ( −x ) + f ( −x ) =⇒ x2f ′ ( −x ) = −xh′ ( x ) ︸ ︷︷ ︸ ≤0 +x · f ( −x ) ≤ x · f ( −x ) ≤ c . ( 9 ) And thus f ′ ( x ) ∈ O ( 1 x2 ) . Theorem 9 ( Error-Bounds of Diff . Sorting Networks ) . If the error of individual conditional swaps of a sorting network is bounded by and the network has ` layers , the total error is bounded by · ` . Proof . For the proof , cf . Supplementary Material C. Discussion . Monotonicity is highly desirable as otherwise adverse effects such as an input requiring to be decreased to increase the output can occur . In gradient-based training , non-mononicity is problematic as it produces gradients with the opposite sign . In addition , as monotonicity is also given in hard sorting networks , it is desirable to preserve this property in the relaxation . Further , monotonic differentiable sorting networks are quasiconvex and quasiconcave as any monotonic function is both quasiconvex and quasiconcave , which leads to favorable convergence rates [ 6 ] . Bounding and reducing the deviation from its hard counterpart reduces the relaxation error , and thus is desirable .

Summary: This work introduces methods to construct differentiable sorting networks that are provably monotonic and have bounded error. To provide such guarantees, the paper derives a set of necessary and sufficient conditions on sigmoid function used for the soft swap operators in the network. It then studies a set of candidate sigmoid functions for such construction, detailing their monotonicity and error guarantees. The experimental results show that the use of monotonic networks can achieve SOTA results when applied to multi-digit ordering on images constructed from MNIST and SVHN.