Rankmax: An Adaptive Projection Alternative to the Softmax Function

This document consists of results that support the material in the paper “Rankmax: An Adaptive Projection Alternative to the Softmax Function”, hereafter referred to as the main paper. It is assumed that the reader is already familiar with the notation and definitions in the main paper. Additional notation. The next quantities are for a closed convex set Z ⊆ R and a proper, lower semicontinuous, and convex function f . Let δZ denote the convex indicator function of Z where δZ(z) = 0 if z ∈ Z and δZ(z) =∞ if z / ∈ Z. Let NZ denote the normal cone of Z, given by NZ(z) := {u ∈ R : 〈u, z̃ − z〉 ≤ 0,∀z̃ ∈ Z} = ∂δZ(z) ∀z ∈ Z. Let ΠZ denote the projection onto the set Z given by ΠZ(z) = argminu∈Z ‖u − z‖/2 for every z ∈ R. Let id denote the identity operator given by id(z) = z for every z ∈ R. Let f∗ denote the convex conjugate of f given by f∗(z) = supz̃∈Rn{〈z̃, z〉 − f(z̃)} for every z ∈ R. A On the equivalence with Bregman projections Recall that the Bregman divergence [3] with a differentiable distance generating function g is given by Dg(x, y) = g(x)− g(y)− 〈∇g(y), x− y〉, and the Bregman projection of a vector z̃ on ∆n−1 k is given by argmin x∈∆n−1 k Dg(x, z̃) = argmin x∈∆n−1 k {g(x)− 〈∇g(z̃), x〉} . (17) This is equivalent to (2) when ∇g(z̃) = αz. This identity is guaranteed to have a solution for all z by strong convexity of g. Indeed by Theorem 23.5 in [8], αz ∈ ∂g(z̃) if and only if z̃ maximizes 〈αz, z̃〉 − g(z), but since g is strongly convex, the latter always has a unique maximizer. Note that the maximization is over all of R, unlike problem (2) where the minimization is over ∆n−1 k . For example, when g(x) = 12‖x‖ 2 2, we have Dg(x, y) = 1 2‖x − y‖ 2 2 and ∇g(x) = x. Thus, (17) and (2) are equivalent with z̃ = αz. When g(x) = ∑n i=1 xi log xi with effective domain {x ∈ R : x ≥ 0}, the function Dg is the (un-normalized) K-L divergence Dg(x, y) = ∑n i=1 xi log(xi/yi) + (yi − xi), defined on R+ × R+, and ∇g(x) = (1 + log xi)i=1.n (note that ∇g is a bijection from R+ to R). Thus (17) and (2) are equivalent up to the change of variable z̃i = eαzi−1. ∗Work done during an internship at Google Research. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. B Properties of pα This section presents several technical results regarding the function pα from the main paper. Before proceeding, we first present the following results on max functions, whose proofs can be found in [5]. Proposition 1. Suppose that for some closed X × Y ⊆ R × R and μ > 0 we have a real–valued function Ψ : R × R 7→ R that satisfies: (A1) −Ψ(x, ·) is a proper, lower semicontinuous, μ–strongly convex function for every x ∈ X; (A2) Ψ(·, y) is continuously differentiable on X for every y ∈ Y ; Moreover, define the functions ψ(x) := max ỹ∈Y Ψ(x, ỹ), y(x) := argmax ỹ∈Y Ψ(x, ỹ), (18) for every x ∈ X . If Y is bounded then: (a) y is continuous on X; (b) ψ is continuously differentiable on X and ∇ψ(x) = ∇xΨ(x, y(x)) for every x ∈ X . We now show Proposition 1 in the main paper, restated below for convenience. Proposition. Suppose g is 1–strongly convex and ∆n−1 k ⊆ dom g. Then the following properties hold for any z ∈ R and α > 0: (a) limt→∞ pt(z) = p∞(z); (b) for any 1 ≤ i, j ≤ n, we have pα(z)i ≥ pα(x)j if and only if zi ≥ zj; (c) the function pα(z) is α–Lipschitz continuous. Proof. For ease of notation, define Ψ(x, y) := 〈x, y〉 − 1 α g(y) ∀(x, y) ∈ R × R, denote ψ and y as in (18), and remark that y = pα. (a) Let {tn} be a positive sequence of scalars tending to infinity and let z ∈ R be fixed. Moreover, denote yn = ptn(z) for every n ≥ 1. Fixing n ≥ 1, the optimality condition of yn is z ∈ 1 αn ∂g(yn) +N∆n−1 k (yn) ⇐⇒ 〈z, z̃ − yn〉 ≤ 1 αn 〈un, z̃ − yn〉 ∀(un, z̃) ∈ ∂g(yn)×∆ k Applying the 1–strong convexity of g to the latter form yields 〈z, z̃ − yn〉 ≤ 1 αn [ g(z̃)− g(yn)− 1 2 ‖z̃ − yn‖ ] , ≤ 1 αn sup z,z̃∈∆n−1 k { g(z̃)− g(z)− 1 2 ‖z̃ − z‖ } } {{ } =:C for every z̃ ∈ ∆n−1 k . Moreover, using the finiteness of g on ∆ n−1 k and the boundedness of ∆ n−1 k it is clear that the quantity C above is finite. Hence, since limn→∞(C/αn) = 0, we conclude that yn converges to a solution in p∞(z). The conclusion now follows from the definitions of pα and p∞. (b) Given a fixed x ∈ R and α > 0, it suffices to prove that pα is variationally monotone, i.e. 〈pα(z)− pα(z̃), z − z̃〉 ≥ 0 for every z, z̃ ∈ R. We first show that ψ is convex. Using the fact that Ψ(·, y) is convex for every y ∈ R, the optimality of y, and Proposition 1(b), we have that for every z, z̃ ∈ R, 0 ≤ Ψ(z̃, y(z))−Ψ(z, y(z))− 〈∇Ψ(z, y(z)), z̃ − z〉 = Ψ(z̃, y(z))− ψ(z)− 〈∇ψ(z), z̃ − z〉 ≤ ψ(z̃)− ψ(z)− 〈∇ψ(z), z̃ − z〉