RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
\ell_{2} versus \ell_{\infty} FGLs
If we compare results from Tables 3 and 7 , we can also find that the discrepancy between matrix product method and sampled lower bound is much smaller in the \ell_{2} case. This could also explain why Gloro works for \ell_{2} -perturbations but not the \ell_{\infty} case in practice, where Leino et al . ( 2021 ) used matrix-norm product to upper bound the Lipschitz constant of the network in Gloro.
Sampling
Sampling can only give a lower bound of the true Lipschitz constant, while we are trying to estimate an upper bound. We use sampling as a sanity check to ensure that the SDP method is at least sound and indeed provides an upper bound of the FGL. It is interesting to see that in networks where we can brute-force enumerate all the activation patterns, sampling provides very close results to the ground-truth ones. Notice that for those networks, there are only a few hidden units (8 or 16), while we sample many (200,000) inputs, which might activate all or most of the patterns. However, for networks with many activation nodes, it is infeasible to have a brute-force enumeration of all the activation patterns, so we do not have the ground-truth information. Sampling has no guarantee whether it can activate all patterns unless we have sampled all possible inputs, which is also impractical.
Multi-layer network guarantees
The discrepancy between the results from sampling and GeoLIP is relatively large for multi-layer networks. The approximation guarantee of GeoLIP is in terms of the FGL, rather than true Lipschitz constant. It is unclear how large the gap between true Lipschitz constant and the FGL is for multi-layer networks. Narrowing this gap is an interesting research direction and beyond the scope of this work. We do not know whether for multi-layer networks, GeoLIP has an approximation guarantee that is independent of the network. We leave this as an open problem.
\ell_{\infty} -FGL estimation of various methods: DGeoLIP and NGeoLIP induce the same values on two layer networks. DGeoLIP always produces tighter estimations than LiPopt and MP do.
Running time (in seconds) of various tools on \ell_{\infty} -FGL estimation: DGeoLIP and NGeoLIP are faster than LiPopt. Notice that the running time is implementation and solver-dependent.
\ell_{\infty} -FGL estimations of different methods for two-layer networks: DGeoLIP and NGeoLIP induce the same estimations, and they are also close to the sampled lower bounds. In the meantime, the result from GeoLIP is tighter than LiPopt’s result.
Average running time (in seconds) of different methods for two-layer-network \ell_{\infty} -FGL estimations: GeoLIPs are faster than LiPopt.
\ell_{\infty} -FGL estimations of different methods for multi-layer networks: GeoLIP’s result is much tighter than the matrix-product method, and LiPopt is unable to handle these networks.
Average running time (in seconds) of GeoLIP for multi-layer-network \ell_{\infty} -FGL estimations.
\ell_{2} -FGL estimations of different methods for two-layer networks: LipSDP and NGeoLIP induce the same estimations, and these results are also close to the sampled lower bounds.
Average running time (in seconds) of LipSDP and NGeoLIP for two-layer-network \ell_{2} -FGL estimations.