Title: Exploiting Dynamic Sparsity in Einsum
Venue: NeurIPS 2025
Accepted: yes
Keywords: einsum

tensor network

dynamic sparsity

tensor contraction
Abstract: Einsum expressions specify an output tensor in terms of several input tensors. They offer a simple yet expressive abstraction for many computational tasks in artificial intelligence and beyond. However, evaluating einsum expressions poses hard algorithmic problems that depend on the representation of the tensors. Two popular representations are multidimensional arrays and coordinate lists. The latter is a more compact representation for sparse tensors, that is, tensors where a significant proportion of the entries are zero. So far, however, most of the popular einsum implementations use the multidimensional array representation for tensors. Here, we show on a non-trivial example that, when evaluating einsum expressions, coordinate lists can be exponentially more efficient than multidimensional arrays. In practice, however, coordinate lists can also be significantly less efficient than multidimensional arrays, but it is hard to decide from the input tensors whether this will be the case. Sparsity evolves dynamically in intermediate tensors during the evaluation of an einsum expression. Therefore, we introduce a hybrid solution where the representation is switched on the fly from multidimensional arrays to coordinate lists depending on the sparsity of the remaining tensors. In our experiments on established benchmark einsum expressions, the hybrid solution is consistently competitive with or outperforms the better of the two static representations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the problem of choosing the correct data structure when evaluating einsum expressions on sparse data, particularly between tensors and coordinate arrays. They show on a family of grid formulae that the latter uses exponentially more flops than the former when contracting the representation of a grid. Then, experiments are performed to empirically support this.\nFurthermore, they propose an hybrid algorithm that switches the representation on the fly, showing positive results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper addresses an important point about the efficiency of the structures we use during long routines of computation, which can lead to dramatic improvement in time and energy consumption.\n- The empirical results are founded in a proved proposition and align with them.The benchmark of einsum instances seems broad and has been recently published.\n- Well written and easy to read.\n\nWeaknesses\n\n- I find it strange that there is no related work. Although the approach seems completely novel, there must be research that speaks about transitions of structures with respect to some threshold. Adding some about this would improve to contextualize the originality of the approach and also to improve my confidence on the score.\n- Some of the limitations could be addressed in the main article such as switching back to a dense format, which should be relatively easy to implement, or experiments on GPUs, which are essential for high performance computation in the context of AI.\n\n\nMinor comments\n\n- Figure 2. Grid Site Length (n) -> Size. Also, why is dense not present on the right subfigure? I think the subfigures should be better explained.\n- Figure 3. Legend missing.\n\nQUESTIONS:\n- Is it possible to apply the hybrid approach to the actual training of a neural network? For example to a simple CNN on MNIST to see the benefits regarding training time and resource use.\n- Are there any related works that discuss the efficiency of structures on neural networks? It would be good to see if other approaches were considered before.\n\nLIMITATIONS:\nYes, in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates the benefits of sparse tensor storage for einsum computations.  \nOn the theoretical side, the authors prove that for a well-chosen family of instances, dense storage can require exponentially more FLOPs for tensor contraction than sparse storage.  \nOn the practical side, they perform an analysis of sparsity evolution as contraction progresses, for a recently-introduced benchmark ([https://benchmark.einsum.org/](https://benchmark.einsum.org/)). This analysis shows that sparsity can sometimes appear halfway through the process, even when the input tensors are dense. Therefore, a new storage strategy is proposed, which switches from dense to sparse when the average density of the tensor sequence falls below a pre-specified threshold. Numerical experiments demonstrate that this strategy can lead to efficiency gains.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Quality\n\n- The separation theorem on model counting for logical formulas is a strong theoretical motivation for sparse storage.\n- The measurement of dynamic sparsity in the Einsum Benchmark provides empirical justification for an adaptive approach.\n\n### Clarity\n\n- The paper is well-written.\n- The experiments are reproducible.\n\n### Significance\n\n- Tensor contraction is a thriving topic in probabilistic inference, and this study opens up several promising research avenues, especially in Section 5.\n\n### Originality\n\n- The focus on CPU operations is unexpected in machine learning but justified for handling dynamic sparsity.\n- The adaptive storage policy is rather naive but it is possible that no one has benchmarked it before.\n\n## Weaknesses\n\n### Quality\n\n- The contraction order used in the experiments is never discussed in the main paper. In the appendix, we only read that “The benchmark includes one contraction order optimized for minimal intermediate size and one for minimal floating-point operations (flops).”\n- The abstract claims that “In our experiments on established benchmark einsum expressions, the hybrid solution is consistently competitive with or outperforms the better of the two static representations.”, but the hybrid algorithm is only compared to its purely dense and sparse counterparts *separately* (Figure 5). It should also be compared to the best of both for each instance.\n\n### Clarity\n\n- It is not clear to me how the hybrid algorithm defines “the average density of *the not-yet contracted tensors*”. After all, a tensor could be involved at several steps of the contraction order (which is defined on axes, if I understand correctly). To put it differently, contractions after step 1 can be defined on intermediate tensors which do not yet exist at the beginning of the process.\n\n### Significance\n\n- In the experiment code, a sparse tensor library called sesum (which I wasn’t able to find elsewhere) is apparently used as a binary wheel, which makes the experiments less reusable and generalizable.\n\nQUESTIONS:\n- To justify a dynamic approach, the authors state that “in einsum expressions involving a large number of tensors, it is not known in advance whether intermediate tensors will become sparse.” Is that obvious? Are there heuristics that could guide a static storage choice?\n- Describe the contraction orders used in the experiments and their impact on the storage and FLOPs requirements.\n- Explain more precisely how the “not-yet contracted tensors” are defined and how their average density is calculated.\n- Add a comparison between the hybrid algorithm and the best of both dense and sparse algorithms, in order to back the claim from the abstract.\n- Defend your choice of sesum as a sparse einsum library, as opposed to the Python sparse library you mention in the introduction.\n\n---\n\nSatisfactory answers to most of these questions can lead me to increase my rating to accept.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a hybrid dense-sparse Einsum for computing large Eisnum expressions. The authors show that there exist non-trivial examples of Einsum programs, in the form of ${\\rm Grid}_n$, where the popular dense implementation is much slower than a coordinate-list-based sparse implementation. Through experiments, the authors show that their proposed hybrid dense-sparse Einsum is much performant that the baseline dense implementation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Simple method that keeps track of the proportion of non-zero elements in tensors; and switch to sparse when below a threshold. This method is simple and performs well.\n- Detailed exposition on the ${\\rm Grid}_n$ tensor network.\n\nWeaknesses:\n- The background is too long in the paper, where the proposed solution appeared at Section 4.4, page 8. The hybrid algorithm is very simple (if non-zero entries >5%, dense). The substance of this paper does not seem to merit a long NeurIPS submission.\n- The proposed method works for ${\\rm Grid}$-type tensor networks, but actual tensor Einsum programs take many different forms (in deep learning; belief propagation; quantum computing; etc). The performance might not generalize to other forms of tensor networks. A good benchmark for this is the Einsum benchmark (https://benchmark.einsum.org/): https://proceedings.neurips.cc/paper_files/paper/2024/file/b1bbfdb9197bfc819a52c34dce493f85-Paper-Datasets_and_Benchmarks_Track.pdf, which I believe that the authors should do experiments on.\n- Tensor operations are now predominantly performed in a parallel way on specialized hardwares such as GPUs or TPUs. Analysis should be done over these heterogeneous hardware.\n\nQUESTIONS:\n- It'd be better to spell out, in pseudocode, the algorithms for contraction for both sparse and dense tensors.\n- Specify what the ${\\rm Grid}$ model is used for. E.g., for modeling magnetic crystal structures? Given more context here would aid understanding of the setting of the method.\n- L147: ${\\rm Grid}_n$: Provide a large illustration / figure of this tensor network.\n- L148: Please add the definition $[n] = \\\\{i \\in \\mathbb{N} | 0 \\le i < n\\\\}$.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper talked about the limitations and constrains in tensor computing, where they modify tensor hypernetworks to speed up the computing process. The proposed method showed a great performance increase across tensors with different densities.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper discussed the constrains and limitations in (sparse) tensor computations.
2. They modified the tensor hypernetworks to speed up the computation. Such modification is suitable for tensors with varied densities.
3. They provide a small C++ module plugs into PyTorch/opt_einsum for CPU capability.

Weaknesses:
1. One limitation I noticed and the author pointed out in the limitation section is how would this method work on GPU. Most tensor computation is utilizing GPU. If the proposed method can only work on CPU. It could greatly limit the applicability of the proposed method.
2. Can the proposed method support sparse to dense fallback?
3. How sufficient is the 5% threshold. How sensitive are results to this value?

QUESTIONS:
Please see previous section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03