Monotonic Differentiable Sorting Networks

Differentiable sorting algorithms allow training with sorting and ranking supervision , where only the ordering or ranking of samples is known . Various methods have been proposed to address this challenge , ranging from optimal transport-based differentiable Sinkhorn sorting algorithms to making classic sorting networks differentiable . One problem of current differentiable sorting methods is that they are non-monotonic . To address this issue , we propose a novel relaxation of conditional swap operations that guarantees monotonicity in differentiable sorting networks . We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic . Monotonicity ensures that the gradients always have the correct sign , which is an advantage in gradient-based optimization . We demonstrate that monotonic differentiable sorting networks improve upon previous differentiable sorting methods . 1 INTRODUCTION . Recently , the idea of end-to-end training of neural networks with ordering supervision by a continuous relaxation of the sorting function has been presented by Grover et al . [ 1 ] . The idea of ordering supervision is that the ground truth order of some samples is known while their absolute values remain unsupervised . This is done by integrating a sorting algorithm in the neural architecture . As for training with a sorting algorithm in the architecture , the error needs to be propagated in a meaningful way back to the neural network , it is necessary to use a differentiable sorting function . Several such differentiable sorting functions have been introduced , e.g. , by Grover et al . [ 1 ] , Cuturi et al . [ 2 ] , Blondel et al . [ 3 ] , and Petersen et al . [ 4 ] . In this work , we focus on analyzing differentiable sorting functions [ 1 ] – [ 4 ] and demonstrate how monotonicity improves differentiable sorting networks [ 4 ] . Sorting networks are a family of sorting algorithms that consist of two basic components : so called “ wires ” ( or “ lanes ” ) carrying values , and conditional swap operations that connect pairs of wires [ 5 ] . An example of such a sorting network is shown in the center of Figure 1 . The conditional swap operations swap the values carried by these wires if they are not in the desired order . They allow for fast hardware-implementation , e.g. , in ASICs , as well as on highly parallelized general-purpose hardware like GPUs . Differentiable sorting networks [ 4 ] continuously relax the conditional swap operations by relaxing their step function to a logistic sigmoid function . One problem that arises in this context is that using a logistic sigmoid function does not preserve monotonicity of the relaxed sorting operation , which can cause gradients with the wrong sign . In this work , we present a family of sigmoid functions that preserve monotonicity of differentiable sorting networks . These include the cumulative density function ( CDF ) of the Cauchy distribution , as well as a function that minimizes the error-bound and thus induces the smallest possible approximation error . For all sigmoid functions , we prove and visualize the respective properties and validate their advantages empirically . In fact , by making the sorting function monotonic , it also becomes quasiconvex , which has been shown to produce favorable convergence rates [ 6 ] . In Figure 2 , we demonstrate monotonicity for different choices of sigmoid functions . As can be seen in Figure 4 , existing differentiable sorting operators are either non-monotonic or have an unbounded error . Following recent work [ 1 ] , [ 2 ] , [ 4 ] , we benchmark our continuous relaxations by predicting values displayed on four-digit MNIST images [ 7 ] supervised only by their ground truth order . The evaluation shows that our method outperforms existing relaxations of the sorting function on the four-digit MNIST ordering task as well as the SVHN ranking task . Contributions . In this work , we show that sigmoid functions with specific characteristics produce monotonic and error-bounded differentiable sorting networks . We provide theoretical guarantees for these functions and also give the monotonic function that minimizes the approximation error . We empirically demonstrate that the proposed functions improve performance . 2 RELATED WORK . Recently , differentiable approximations of the sorting function for weak supervision were introduced by Grover et al . [ 1 ] , Cuturi et al . [ 2 ] , Blondel et al . [ 3 ] , and Petersen et al . [ 4 ] . In 2019 , Grover et al . [ 1 ] proposed NeuralSort , a continuous relaxation of the argsort operator . A ( hard ) permutation matrix is a square matrix with entries 0 and 1 such that every row and every column sums up to 1 , which defines the permutation necessary to sort a sequence . Grover et al . relax hard permutation matrices by approximating them as unimodal row-stochastic matrices . This relaxation allows for gradient-based stochastic optimization . On various tasks , including sorting four-digit MNIST numbers , they benchmark their relaxation against the Sinkhorn and Gumbel-Sinkhorn approaches proposed by Mena et al . [ 8 ] . Cuturi et al . [ 2 ] follow this idea and approach differentiable sorting by smoothed ranking and sorting operators using optimal transport . As the optimal transport problem alone is costly , they regularize it and solve it using the Sinkhorn algorithm [ 9 ] . By relaxing the permutation matrix , which sorts a sequence of scalars , they also train a scalar predictor of values displayed by four-digit numbers while supervising their relative order only . Blondel et al . [ 3 ] cast the problem of sorting and ranking as a linear program over a permutahedron . To smooth the resulting discontinuous function and provide useful derivatives , they introduce a strongly convex regularization . They evaluate the proposed approach in the context of Top-k classification and label ranking accuracy via a soft Spearman ’ s rank correlation coefficient . Recently , Petersen et al . [ 4 ] proposed differentiable sorting networks , a differentiable sorting operator based on sorting networks with differentiably relaxed conditional swap operations . Differentiable sorting networks achieved a new state-of-the-art on both the four-digit MNIST sorting benchmark and the SVHN sorting benchmark . Applications and Broader Impact . In the domain of recommender systems , Lee et al . [ 10 ] propose differentiable ranking metrics , and Swezey et al . [ 11 ] propose PiRank , a learning-to-rank method using differentiable sorting . Other works explore differentiable sorting-based top-k for applications such as differentiable image patch selection [ 12 ] , differentiable k-nearest-neighbor [ 1 ] , [ 13 ] , top-k attention for machine translation [ 13 ] , and differentiable beam search methods [ 13 ] , [ 14 ] . 3 BACKGROUND : SORTING NETWORKS . Sorting networks have a long tradition in computer science since the 1950s [ 5 ] . They are highly parallel data-oblivious sorting algorithms . They are based on so-called conditional pairwise swap operators that map two inputs to two outputs and ensure that these outputs are in a specific order . This is achieved by simply passing through the inputs if they are already in the desired order and swapping them otherwise . The order of executing conditional swaps is independent of the input values , which makes them data-oblivious . Conditional swap operators can be implemented using only min and max . That is , if the inputs are a and b and the outputs a′ and b′ , a swap operator ensures a′ ≤ b′ and can easily be formalized as a′ = min ( a , b ) and b′ = max ( a , b ) . Examples of sorting nets are the odd-even network [ 15 ] , which alternatingly swaps odd and even wires with their successors , and the bitonic network [ 16 ] , which repeatedly merges and sorts bitonic sequences . While to odd-even network requires n layers , the bitonic networks uses the divide-and-conquer principle to sort within only ( log2 n ) ( 1 + log2 n ) /2 layers . Note that , while they are similar in name , sorting networks are not neural networks that sort . 3.1 DIFFERENTIABLE SORTING NETWORKS . In the following , we recapitulate the core concepts of differentiable sorting networks [ 4 ] . An example of an odd-even sorting network is shown in the center of Figure 1 . Here , odd and even neighbors are conditionally swapped until the entire sequence is sorted . Each conditional swap operation can be defined in terms of min and max as detailed in Section 3 . These operators can be relaxed to differentiable min and max . Note that we denote the differentiable relaxations in italic font and their hard counterparts in roman font . Note that the differentiable relaxations min and max are different from the commonly used softmin and softmax , which are relaxations of argmin and argmax [ 17 ] . One example for such a relaxation of minσ and maxσ is the logistic relaxation minσ ( a , b ) = a · σ ( b− a ) + b · σ ( a− b ) and maxσ ( a , b ) = a · σ ( a− b ) + b · σ ( b− a ) ( 1 ) where σ is the logistic sigmoid function with inverse temperature β > 0 : σ : x 7→ 1 1 + e−βx . ( 2 ) Any layer of a sorting network can also be represented as a relaxed and doubly-stochastic permutation matrix . Multiplying these ( layer-wise ) permutation matrices yields a ( relaxed ) total permutation matrix P. Multiplying P with an input x yields the differentiably sorted vector x̂ = Px , which is also the output of the differentiable sorting network . Whether it is necessary to compute P , or whether x̂ suffices , depends on the specific application . For example , for a cross-entropy ranking / sorting loss as used in the experiments in Section 6 , P can be used to compute the cross-entropy to a ground truth permutation matrix Q . In the next section , we build on these concepts to introduce monotonic differentiable sorting networks , i.e. , all differentiably sorted outputs x̂ are non-decreasingly monotonic in all inputs x . 4 MONOTONIC DIFFERENTIABLE SORTING NETWORKS . In this section , we start by introducing definitions and building theorems upon them . In Section 4.2 , we use these definitions and properties to discuss different relaxations of sorting networks . 4.1 THEORY . We start by defining sigmoid functions and will then use them to define continuous conditional swaps . Definition 1 ( Sigmoid Function ) . We define a ( unipolar ) sigmoid ( i.e. , s-shaped ) function as a continuous monotonically non-decreasing odd-symmetric ( around 12 ) function f with f : R→ [ 0 , 1 ] with lim x→−∞ f ( x ) = 0 and lim x→∞ f ( x ) = 1 . Definition 2 ( Continuous Conditional Swaps ) . Following [ 4 ] , we define a continuous conditional swap in terms of a sigmoid function f : R→ [ 0 , 1 ] as minf ( a , b ) : = a · f ( b− a ) + b · f ( a− b ) max f ( a , b ) = a · f ( a− b ) + b · f ( b− a ) ( 3 ) argminf ( a , b ) = ( f ( b− a ) , f ( a− b ) ) argmax f ( a , b ) = ( f ( a− b ) , f ( b− a ) ) . ( 4 ) We require a continuous odd-symmetric sigmoid function to preserve most of the properties of min and max while also making argmin and argmax continuous as shown in Supplementary Material B . In the following , we establish doubly-stochasticity and differentiability of P , which are important properties for differentiable sorting and ranking operators . Theorem 3 ( Doubly-Stochasticity and Differentiability of P ) . ( i ) The relaxed permutation matrix P , produced by a differentiable sorting network , is doubly-stochastic . ( ii ) P has the same differentiability as f , e.g. , if f is continuously differentiable in the input , P will be continuously differentiable in the input to the sorting network . If f is differentiable almost everywhere ( a.e . ) , P will be diff . a.e . Proof . ( i ) For each conditional swap between two elements i , j , the relaxed permutation matrix is 1 at the diagonal except for rows i and j : at points i , i and j , j the value is v ∈ [ 0 , 1 ] , at points i , j and j , i the value is 1− v and all other entries are 0 . This is doubly-stochastic as all rows and columns add up to 1 by construction . As the product of doubly-stochastic matrices is doubly-stochastic , the relaxed permutation matrix P , produced by a differentiable sorting network , is doubly-stochastic . ( ii ) The composition of differentiable functions is differentiable and the addition and multiplication of differentiable functions is also differentiable . Thus , a sorting network is differentiable if the employed sigmoid function is differentiable . “ Differentiable ” may be replaced with any other form of differentiability , such as “ differentiable a.e. ” Now that we have established the ingredients to differentiable sorting networks , we can focus on the monotonicity of differentiable sorting networks . Definition 4 ( Monotonic Continuous Conditional Swaps ) . We say f produces monotonic conditional swaps if minf ( x , 0 ) is non-decreasingly monotonic in x , i.e. , min ′f ( x , 0 ) ≥ 0 for all x . It is sufficient to define it w.l.o.g . in terms of minf ( x , 0 ) due to its commutativity , stability , and odd-symmetry of the operators ( cf . SM ) . Theorem 5 ( Monotonicity of Continuous Conditional Swaps ) . A continuous conditional swap ( in terms of a differentiable sigmoid function f ) being non-decreasingly monotonic in all arguments and outputs requires that the derivative of f decays no faster than 1/x2 , i.e. , f ′ ( x ) ∈ Ω ( 1 x2 ) . ( 5 ) Proof . We show that Equation 5 is a necessary criterion for monotonicity of the conditional swap . Because f is a continuous sigmoid function with f : R→ [ 0 , 1 ] , minf ( x , 0 ) = f ( −x ) · x > 0 for some x > 0 . Thus , montononicity of minf ( x , 0 ) implies lim supx→∞minf ( x , 0 ) > 0 ( otherwise the value would decrease again from a value > 0 . ) Thus , lim x→∞ minf ( x , 0 ) = lim x→∞ f ( −x ) · x = lim x→∞ f ( −x ) 1/x ( L ’ Hôpital ’ s rule ) = lim x→∞ −f ′ ( −x ) −1/x2 ( 6 ) = lim x→∞ f ′ ( −x ) 1/x2 = lim x→∞ f ′ ( x ) 1/x2 = lim sup x→∞ f ′ ( x ) 1/x2 > 0⇐⇒ f ′ ( x ) ∈ Ω ( 1 x2 ) . ( 7 ) assuming limx→∞ f ′ ( x ) 1/x2 exists . Otherwise , it can be proven analogously via a proof by contradiction . Corollary 6 ( Monotonic Sorting Networks ) . If the individual conditional swaps of a sorting network are monotonic , the sorting network is also monotonic . Proof . If single layers g , h are non-decreasingly monotonic in all arguments and outputs , their composition h ◦ g is also non-decreasingly monotonic in all arguments and outputs . Thus , a network of arbitrarily many layers is non-decreasingly monotonic . Above , we formalized the property of monotonicity . Another important aspect is whether the error of the differentiable sorting network is bounded . It is very desirable to have a bounded error because without bounded errors the result of the differentiable sorting network diverges from the result of the hard sorting function . Minimizing this error is desirable . Definition 7 ( Error-Bounded Continuous Conditional Swaps ) . A continuous conditional swap has a bounded error if and only if supxminf ( x , 0 ) = c is finite . The continuous conditional swap is therefore said to have an error bounded by c. It is sufficient to define it w.l.o.g . in terms of minf ( x , 0 ) due to its commutativity , stability , and odd-symmetry of the operators ( cf . SM ) . In general , for better comparability between functions , we assume a Lipschitz continuous function f with Lipschitz constant 1 . Theorem 8 ( Error-Bounds of Continuous Conditional Swaps ) . ( i ) A differentiable continuous conditional swap has a bounded error if f ′ ( x ) ∈ O ( 1 x2 ) . ( 8 ) ( ii ) If it is additionally monotonic , the error bound can be found as limx→∞minf ( x , 0 ) and additionally the error is bound only if Equation 8 holds . Proof . ( i ) W.l.o.g . we consider x > 0 . Let g ( z ) : = f ( −1/z ) , g ( 0 ) = 0 . Thus , g′ ( z ) = 1/z2 · f ′ ( −1/z ) ≤ c according to Equation 8 . Thus , g ( z ) = g ( 0 ) + ∫ z 0 g′ ( t ) dt ≤ c · z . Therefore , f ( −1/z ) ≤ c · z =⇒ 1/z · f ( −1/z ) ≤ c and with x = 1/z =⇒ x · f ( −x ) = minf ( x , 0 ) ≤ c. ( ii ) Let minf ( x , 0 ) be monotonic and bound by minf ( x , 0 ) ≤ c. For x > 0 and h ( x ) : = minf ( x , 0 ) , h′ ( x ) = −x · f ′ ( −x ) + f ( −x ) =⇒ x2f ′ ( −x ) = −xh′ ( x ) ︸ ︷︷ ︸ ≤0 +x · f ( −x ) ≤ x · f ( −x ) ≤ c . ( 9 ) And thus f ′ ( x ) ∈ O ( 1 x2 ) . Theorem 9 ( Error-Bounds of Diff . Sorting Networks ) . If the error of individual conditional swaps of a sorting network is bounded by and the network has ` layers , the total error is bounded by · ` . Proof . For the proof , cf . Supplementary Material C. Discussion . Monotonicity is highly desirable as otherwise adverse effects such as an input requiring to be decreased to increase the output can occur . In gradient-based training , non-mononicity is problematic as it produces gradients with the opposite sign . In addition , as monotonicity is also given in hard sorting networks , it is desirable to preserve this property in the relaxation . Further , monotonic differentiable sorting networks are quasiconvex and quasiconcave as any monotonic function is both quasiconvex and quasiconcave , which leads to favorable convergence rates [ 6 ] . Bounding and reducing the deviation from its hard counterpart reduces the relaxation error , and thus is desirable .

Summary: This paper introduces monotic differentiable sinkhorn networks. Authors argue that mononicity is a desired property because it leads to better bounds in errors. Authors show their proposed approach beats the state of the art on sorting MNIST and SVHN digits