A Gradient Accumulation Method for Dense Retriever under Memory Constraint
Authors: Jaehee Kim
Yukyung Lee
Pilsung Kang
Venue: NeurIPS 2024 poster (2024)
Keywords: Dense Retriever
Efficient Training
Memory Reduction
Memory Bank
Dual Encoder
Primary area: natural_language_processing
TL;DR: Our proposed method (ContAccum) reduces hardware needs for large batch training in information retrieval, outperforming not only memory reduction methods but also high-resource scenario in low-resource settings with stable dual encoder training.

Abstract: InfoNCE loss is commonly used to train dense retriever in information retrieval tasks. It is well known that a large batch is essential to stable and effective training with InfoNCE loss, which requires significant hardware resources. Due to the dependency of large batch, dense retriever has bottleneck of application and research. Recently, memory reduction methods have been broadly adopted to resolve the hardware bottleneck by decomposing forward and backward or using a memory bank. However, current methods still suffer from slow and unstable train. To address these issues, we propose Contrastive Accumulation (ContAccum), a stable and efficient memory reduction method for dense retriever trains that uses a dual memory bank structure to leverage previously generated query and passage representations. Experiments on widely used five information retrieval datasets indicate that ContAccum can surpass not only existing memory reduction methods but also high-resource scenarios. Moreover, theoretical analysis and experimental results confirm that ContAccum provides more stable dual-encoder training than current memory bank utilization methods.

A Gradient Accumulation Method for Dense Retriever under Memory Constraint
Jaehee Kim!
Yukyung Lee²
Pilsung Kang'*
Seoul National University Boston University {j jjaehee_kim, pilsung_kan (angs_knn]]asa @snu. ac ac.kr ylee50bu. edu
Abstract
InfoNCE loss is commonly used to train dense retriever in information retrieval tasks. It is well known that a large batch is essential to stable and effective training with InfoNCE loss, which requires significant hardware resources. Due to the dependency of large batch, dense retriever has bottleneck of application and research. Recently. memory reduction methods have been broadly adopted to resolve the hardware bottleneck by decomposing forward and backward or using memory bank. However, current methods still suffer from slow and unstable training. To address these issues, we propose Contrastive Accumulation (CONtAccUM), stable and efficient memory reduction method for dense retriever trains that uses a dual memory bank structure to leverage previously generated query and passage representations. Experiments on widely used five information retrieval datasets indicate that CONTACCUM can surpass not only existing memory reduction methods but also high- sigh-resource scenario. Moreover, theoretical analysis and experimental results confirm that CONTACCUM provides more stable dual-encoder training than current memory bank utilization methods.
1 Introduction
Dense retriever aims to retrieve relevant passages from database in response to user queries with neural networks (43). Karpukhin al. [16] and Lee al. 220 introduced the in-batch negative sampling for training dense retriever with InfoNCE loss [36]. where relevant passages from other queries the same batch utilized as negative passages. This negative sampling strategy has been widely adopted in subsequent dense retriever studies, including supervised retriever [16] 31
28 41
retriever pre- pre-training 7 24 12 phrase retriever 19 25 and generative retriever
Training dense retriever with InfoNCE loss drives the representations of queries and relevant passages closer and pushes the representations of unrelated passages apart, which can be seen as form of metric learning
17
Many dense retriever methodologies utilize large batch to incorporate more negative samples 41 28 [1). Theoretically, it has been demonstrated that more negative samples in InfoNCE loss lead to tighter lower bound on mutual information between query and passage [36] Empirical studies have
shown that the dense retriever performs better with large batch [28] 43 42) However, training with
large batches requires high-resource, posing a challenge for dense retriever research and applications.
line of research has focused on overcoming these limitations by approximating the effects of large batch sizes. Gradient Accumulation (GradAccum), common method for approximating large batch, reduces memory usage by splitting the large batch into smaller batches. However, GradAccum has limitations in the context of InfoNCE loss because it reduces negative samples per query by the smaller
indicates corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

batch 19]. To overcome the limitation of GradAccum, Gao et al. [9] proposed the Gradient Cache (GradCache), which approximates large batch by decomposing the backpropagation process and adapts additional forwarding process for calculating gradients. However, GradCache has limitations, including significant additional training time due to computational overhead and the inability to surpass high-resource scenario where accelerators are sufficient to train large batch. Additionally, pre-batch negatives [19] caches passage representations from previous steps to secure additional negative samples, but it also shows unstable train and marginal performance gain.
In this study, we propose Contrastive Accumulation (CoNTACcUM), which demonstrates high performance and stable training under memory constraints. CONTACCUM leverages previously generated query and passage representations through a memory bank, enabling the use of more negative samples. Our analysis of the gradients reveals that utilizing a memory bank for both query and passage leads to stable training. The specific contributions of this study are as follows:
We propose CONTACCUM, method utilizing dual memory bank strategy that can outperform not only existing memory reduction methods but also high-resource scenario in low-resource setting.
We show that our method is time efficient, reducing the training time compared to existing memory reduction methods.
We demonstrate the cause of training instability in existing memory bank utilization methods through mathematical analysis and experiments, showing that the dual memory bank strategy stabilizes training.
2
Related works
gradient
partial gradient
gradlent
query representation
passage representation
(a) Gradient Cache
(b) Gradient Accumulation
(c) Contrastive Accumulation
Figure 1: Illustrations of CONTACCUM and Comparative Methods. The illustrations show total
batch size (Niota) of 4, a local batch size (Nocat) of 2, and a memory bank size (Nmemory 4. (a) GradCache uses Nevul negative passages. (b) GradAccum uses Niccal 1 negative passages. (c)
CONTACCUM leverages Nicall + Nmemory negative samples, more than Nuvtal 1.
2.1 Memory reduction in information retrieval
GradAccum is the most common method to address memory reduction problem. By using GradAccum, gradients of the total batch can be stored by sequentially processing local batches through forward and backward passes, even when the total batch cannot be processed at once. However, shown in Figure (b)) GradAccum is not a proper memory reduction method for the in-batch negatives, as uses fewer negative samples than the total batch. We will discuss the limitation of GradAccum for contrastive learning in detail in subsection 3.1
GradCache reduces memory usage in contrastive learning by decomposing the backpropagation process. Specifically, as shown in Figure [ (a), calculates the loss without storing activations during the forward pass using the total batch. Then, it computes and stores the gradient from the loss to the representations. Next, it performs additional forward passes for the local batch to store activations and sequentially calculates gradients from each representation to the model weights. This allows GradCache to use the same number of negative samples as the total batch, approximating the performance of the total batch. However, GradCache cannot surpass the performance of high-resouree

---

scenario because it uses the same number of negative samples. Also, GradCache requires significant amount of time due the complex forward and backward processes.
2.2 Memory bank
The memory bank structure for metric learning was initially proposed for the vision domain, where stores representations generated by the encoder previous batches [40 [9]. Combined with the NCE loss [T0,, memory bank structures have been widely used to train uni-encoder vision models [1 3 38 However, directly adapting this approach to information retrieval tasks, where a dual-encoder structure is commonly used, is challenging. This is due to several factors: In multi-modal settings, Li ct al. [22 21 have employed momentum encoders for both image and text modalities to generate cached representations. However, these approaches do not directly address the asymmetric nature information retrieval, where the goal is to retrieve relevant passages for a given query rather than retrieving relevant queries for given passage.
In the information retrieval task, Izacard et al. [12] proposed caching representations generated by a momentum encoder [1I, but they only consider the uni-encoder setting. Lee et al. [19] introduced pre-batch negatives that extend the number of negative samples by caching passage representations with memory bank in dual-encoder setting. However, pre-batch negatives final few epochs of the training process due to the rapid changes in encoder representations early in training. which can cause instability when using a memory bank [38 37).
applied only in the
In summary, existing dense retrievers depend on in-batch negative sampling. necessitating large batch sizes and costly hardware settings. While memory reduction methods have been studied to address this, they often result in slower training or unstable training. Therefore, we propose CONTACCUM, memory reduction method designed to ensure fast and stable training of dense retrievers.
3 Proposed Method
3.1 Preliminary: InfoNCE loss with GradAccum
Before introducing our method, we first examine GradAccum with InfoNCE loss. Karpukhin et al. [16] proposed training method for dense retriever using InfoNCE loss. With a batch size N, dense retrievers are trained by minimizing the negative log-likelihood over all query representations (Q) and passage representations. Specifically, they utilized in-batch negative sampling (P) in the same batch for efficiency, encoded by the query and passage encoders as:
N
exp(Sk.9/T)
C(S) = N
log
where 5 = Softmax PT)
E NNN
(I)
exp(S(a))7))
The in-batch negative sampling efficiently obtains N 1 negative passages per query from relevant passages of other queries, as shown in Equation [ Consequently, the number of negative passages increases with a larger batch size. Due to this characteristic of in-batch negative sampling, dense
retriever is trained using extremely large batch size, ranging from 128 to 8192 [6 12 28 5
22.. However, the need to process all data in memory simultaneously requires multiple high-cost accelerators, ranging from [T6] to 32 12] This creates a hardware bottleneck that constrains various research and applications.
In low-resource setting, GradAccum is employed train models with the total batch size (Noota) which cannot be fitted in the limited memory. GradAccum decomposes the total batch into accumulation steps, K, and processes the local batch, Nivall Nusal/ K, through forward and backpropagation K times to calculate gradients. The process of computing InfoNCE Loss with GradAccum is as follows.
First, the query, and document, p. are encoded by the query encoder, fé. and passage encoder, g training step / respectively:
q' fg( E Edintaa p' - (p( E IR Iudns
(2)
where dmnssee denotes the dimension of query and passage representation. The query encoder, f. and passage encoder, g. parameterized by O and A respectively. The query and passage representations
3

---

Mumory Bank
MP
Manery Bark
M
==
P
P
2=
Pi
P
Macuary
Maresry
par gradient
Marrey
Memory Burn
gradient
Enqueue
Mael
Mosa
Nanuery
Nacal
Moek
Figure 2: Training process of CONTACCUM at each accumulation step. The illustration shows total batch size (Nosal of 4, an accumulation step (K) of 2, and a memory bank size (N (Lmmmmory) of 4. The dual memory bank caches both query and passage representations. New representations enqueued, and the oldest are dequeued at each step, maintaining the similarity matrix (S₂) size at (Niocat Mmemory, Nivcal Nmemory).
within the same local batch at the k-th accumulation step are given follows:
Q {qi ..... Aiv...a } I. ng-a doodel
P {pi, Ppi...l IR Neu X dootel
Using Equation D the loss for the k-th accumulation step is calculated, and the loss for the total batch used for one weight update is obtained as shown in Equation
C h= -
E ((S)), where Sy Softmax Q (P() R-Nica NNvo
In Equation 4 the number of negative passages in each accumulation step is Niccal 1, which is fewer than the number of negative passages when using the total batch, Nootl - 1. This reduction in the number of negative samples results from that GradAccum pass. Consequently, GradAccum cannot maintain the number of negative passages in low-rew low-resource setting, while the total amount of data used for weight updates the same as the total batch.
Nical passages in single forward
3.2 CONTACCUM
To address the issue of fewer negative passages being used with GradAccum, we propose CONTACCUM, method that utilizes dual memory bank structure to cache representations for both queries and passages. The query and passage memory banks (Mqu Mp) are implemented as First-In-First- queues storing memory and N representations respectively. For example, as shown in Figure 2 the oldest representations in the memory bank (Pt-1, - are replaced with the newly-generated ones (Pf, Q9). ). Memory bank strategy is computationally efficient as reuses generated representations from previous iterations (37 38 19]. Unlike Lee et al. [19), which only
utilized passage memory bank Mp. CONTACCUM employs a dual memory bank by also utilizing a query memory bank Mq.
CONTACCUM constructs the similarity matrix using both current and stored representations from the dual memory bank as illustrated in Figure 2 It is equivalent to modifying S₁ in Equation as:
Q Q U sg (gg( e (Nooe No menurry daudd
(5)
= P Ugg(Mp) E TR. (Nea N davadd
(6)
S₁ = Softmax Q Q
The backpropagation process using InfoNCE loss proceeds in the same manner as in Equation However, since the representations in the memory bank do not have stored activations by the stop-gre coo-ggraieet operation(sg() .), the gradients are not pack-propagated through the representations in the memory bank.
The number of negative passages CONTACCUM N + NP Narmorr which is greater than GradAccum. Furthermore, if NiRe > Niocal x (K- CONTACCUM can utilize more negative passages than the total batch, enabling superior performance low-resource setting compared to high-resource scenario.

---

3.3 Gradient analysis with dual memory bank
We analyze the InfoNCE loss backpropagation process in information retrieval tasks, extending
the analysis by Gao et al. [9] to consider using the memory bank. In the partial derivatives of the loss function with respect to the encoders, Ve[(S&) Eqeaq
8C(S)) dq:
VA((Sk)
deg
8e
EpTept
OC(S.)
Jpi
the partial derivative terms for each representation are given by: DA
Opi
Nocar Mansey
aC(S))
(pr
Σ
SR(I.) Pÿ))
(8)
öqi
Nocal + N
memory
Nust eaenoyy
AL(S)) 0p: pocal + No
q
Σ
Sk(B,) q3),
(9)
Memorr
where JR((J) denotes the similarity between i-th query and j-th passage in the similarity matrix Sk. of the k-th accumulation step. Detailed differentiation steps are provided in Appendix 6
Equations 8 and 9 have a similar structure, indicating that the gradients of the two encoders are influenced by the representations generated by the opposite encoder. The difference lies in the summation targets, which are determined by the size of the memory banks, The gradient calculation for the query encoder uses Niccal + Memorry passage representations, while the passage encoder uses memory query representations.
Nicca + No
Pre-batch negatives only leverages the passage memory bank where Aimemory N memoory 0. The tendency where [[ < ||VAC(S&)) is caused by the difference in the number of representations used for the gradient calculations of the encoders. In dual-encoder training, if the gradient norms of the two encoders remain imbalanced, the encoder with the larger gradient norm converges faster, making balanced training challenging [4] 33 Therefore, the unstable training with a memory bank is caused not only by rapid changes in encoder representations 37 38 but also by the difference in the gradient norms between the dual-encoders. We refer this problem as the gradient norm imbalance problem.
The gradient norm imbalance problem can be resolved by using memory banks of equal size for queries and passages, Nmmmory = Nimemory =
Nmemory- This ensures that the gradient norms of the two encoders remain similar and stabilizes the training process. Further analysis is provided in Sections 5.2 and 5.5
Experimental setups
Resources. All experiments were conducted on single A100 80GB GPU. For high-resource scenario, we considered situations where 80GB of memory is available. For low-resource settings. we assumed available memory as widely used commercial GPUs: 1IGB (GTX-I080T)) 24GB (RTX-3080T) RTX-4090TT). To ensure strict experimental conditions, we used a function from the PyTor [27] to limit the available memory Unless otherwise stated, all experiments assumed low resource setting where only 11GB memory available.
Datasets and evaluation metrics. The datasets used for the experiments were Natural Questions (NQ) 18). TriviaQA (15), Curated TREC (TREC) (. and Web Questions (WebQ) 2 processed by DPR and MS Marco 26 For Natural Questions, TriviaQA, Curated TREC, and Web Questions, we used the preprocessed data provided by DPR [16], which includes hard negative samples, positive passages, and answer annotations. Only queries with both positive and hard negative passages were used for training. For MS Marco, we utilized the preprocessed data from BEIR (35] and filtered BM25 [22 hard negatives using cross-encoder scores from the sentence-tramee transformers library [30]. Specifically, we considered passages as hard negatives if their cros-encoder scores were at least points higher than the positive passages' scores, following the preprocessing pipeline provided by centence-transforee
For evaluation metrics, Top@k was used for Natural Questions, TriviaQA, TREC, and WebQ following DPR. Also, we evaluate MS Marco using NDCG@K K and Recall K, widely used metrics
Using the torch. cuda. set_pett hhtpererrres__ereeeeeeeeeeeeeeee process_memory.......o....popoee fraction function in Porch allows for restricting the memory used during training. regardless of the total available memory.
5

---

for dense retriever. NQ and TriviaQA were evaluated using test sets, while TREC, WebQ, and MS Marco were evaluated using dev sets. Additionally, the entire document set was used for evaluation.
Implementation details. The experimental code was adapted from nano-DPk which provides simplified training and evaluation pipeline for DPR. All experiments were conducted using the BERT 6 model. To maintain consistency with DPR's experimental setup, NQ and TREC were trained for 40 epochs, and TriviaQA and WebQ for 100 epochs. For MS Marco, performance saturated at 10 epochs, so it was trained for 10 epochs. Other training settings were also kept consistent with DPR. Detailed settings are provided in Appendix 6
The optimal memory bank size, N Nmemory. was selected using evaluation data with candidates [128, 512, 2048], resulting in 2,048 for NQ and 512 for TriviaQA. For MS Marco, WebQ. and TREC, due to the lack of evaluation data, Nmemory were set based on dataset size: 1,024 for MS Marco, and 128 for WehQ and TREC.
Baselines. We established three baselines for each scenario, and all methods were trained with hard negatives. First, we reported the performance of DPR with the maximum batch size possible for each scenario. Further, we reported the performance of GradAccum with the total batch size of Nova 128. The local batch size Niecll varied by the scenario with K / We also conducted experiments with GradCache [9]. known for approximating total batch performance, using the same Nical for single forwarding.
Experimental results
5.1 Performance across different resource constraints
Table 1: Performance of different methods in low-resource settings (11GB, 24GB) and high-resource
(80GB) setting. In the high-resource setting, the score of the original DPR [16] paper (original) and
the reproduced implementation (implemented) are listed. The best score for each training environment
bolded, and scores surpassing the high-resource setting are marked with N₁ denotes the local
batch size Mocal, N₂ denotes the total batch size No and K represents the accumulation step.
Batch Size
MS Mareo
NQ
TriviaQA
WebQ
TREC
Method
NJKIN,
Recall
Top
Top
Top
Top
20
100
20
100
20
100
20
100
20
100
20
100
VRAM=IIGB
27.9
23.5
8.3
15.2
72.2
81.5
73.7
81.9
72.5
81.4
80.8
88.9
GradAccum
8/16/128
31.1
26.4
10.1
18.1
77.1
84.7
78.4
84.8
74.6
81.9
79.7
89.9
GradCache
8/16/128
34.9
30.6
12.8
22.4"
79.5"
85.9
79.4
85.1
75.1"
82.3
81.6
90.2
CONTACCUM (ours)
8/16/128
39.1
32,9"
14.4"
23.8"
80.1"
86.5
79.8
85.3
75.4"
82.1
83.3"
90.5
VRAM-24GR 24GB
DPR
32/1/ 32
33.1
28.5
11.5
19.6
77.0
84.8
77.5
84.2
74.8"
82.1
82.7'
89.8
GradAccum
28.2
11.8
79.3
GradCache
32/4/128
35.5'
31.0"
12.8
22.1
79.6"
86.0
79.7'
85.1
74.7
81.8
81.3
89.6
CONTACCUM (ours)
32/4/128
39.0"
32.9"
14.6
24.1"
800.6"
86.3"
79.4
85.1
75.0"
82.5"
$1.8
89.5
VRAM=80GB
DPR (implemented)
128/1/128
35.1
30.8
12.7
79.5
85.1
74.7
90.5
DPR (original)
128/1/128
78.4
85.4
79.4
85.0
73.2
81.4
79.8
89.1
CONTACCUM outperforms the high-resource DPR even under low-resource constraints. Table [ compares the performance of CONTAcCUMwith baseline methods under low-resource setting. Notably, CONTACCUM, with only 1IGB of memory, surpasses the performance of DPR in the high-resource setting (80GB). This demonstrates that CONTACCUMiS not only memory-efficient but also achieves superior performance compared to the baseline.
CONTACCUM maintains consistent performance across different memory constraints. CONTACCUM exhibits robust performance regardless of the memory constraint level (IIGB or 24GB), with only minor variations between the two settings. In contrast, the performance of both DPR and GradAccum improves as the available memory increases from 1GB to 24GB. This suggests
p httggghhhh..om/amammmmaddlnnnnn "bert-base uncased

---

that the performance gains of CONTACCUM are not significantly affected by the severity of memory limitations,
The effectiveness of CONTACCUM is amplified under more severe memory constraints. While CONTACCUM consistently outperforms the baseline methods in both 11GB and 24GB scenarios, the performance gap between CONTAcCUM and the baselines is more substantial in the 1IGB setting. This indicates that the advantages of CONTACCUM are particularly evident when memory constraints are stringent, emphasizing its effectiveness in low-resource setting. The strong performance of CONTACCUM can be attributed to its dual memory bank strategy, which allows it utilize more negative samples than GradCache, even in low-resource settings. Furthermore, CONTACCUM outperforms the high-resource setting in 18 out of 24 metrics, improving up 4.9 points. In contrast, GradCache only surpasses the high-resource setting in 8 metrics, with marginal improvements likely due to randomness. These results demonstrate the fundamental advantage of CONTACCUM in achieving superior performance compared to both the baselines and the high-resource setting.
5.2 Influence of each components in CONTACCUM
Table 2: Results of removing the components of CONTACCUM, The DPR performance in low-resource (BSZ=8) and high-resource (BSZ=128) settings are shown baselines. The best-performing method is highlighted in bold.
w/ Hard Negative
w/o Hard Negative
Method
Top@20
Method
Top@20
DPR (BSZ=8)
70.9
DPR (BSZ=8)
63.7
DPR (BSZ=128)
78.4
DPR (BSZ=128)
74.3
CONTACCUM (ours)
78.8
CONTAcCUM (ours)
76.3
w/o. M,
70.8
w/o, M,
72.3
w/o. Past Enc.
76.5
w/o. Past Enc.
73.4
w/o. M,st Enc.
67.8
w/o. M,y/Past Enc.
73.9
w/o. GradAccum
76.7
w/o. GradAccum
74.1
Table 2 shows the influence of key components in CONTACCUM by removing each component with NQ. We also reported experiments that excluded hard negatives during training observe the tendency. The most significant performance drop occurred when the query memory bank M, was removed, indicating its crucial role in CONTACCUM. The other components of CONTACCUM also contributed to the overall performance, with consistent trends regardless of using hard negatives.
Passage memory bank alone degrades performance due to gradient norm imbalance. Specifically, using only the passage memory bank (w/o. M2,,, similar to the pre-batch negatives, led to an 8-point performance drop in Top@2 20 compared to CONTACCUM. This decrease can be attributed to the gradient norm imbalance problem highlighted in Section 3.3 Section 5.5 further analyzes this issue.
GradAccum and past encoder representations crucial for stable training and performance. Moreover, when GradAccum was not applied (w/o. GradAccum), a 2.1-nt performance decline was observed in Top @op@22 20, highlighting the importance of involving more data in gradient calculations for stable training in CONTACCUM. Additionally, a 2.3-point performance decrease was noted when representations generated by past encoders were not used (w/o. Past Enc.). This finding confirms that past encoder representations contribute to training, as suggested by previous studies 37 38 19 However, unlike pre-batch negatives, query memory bank M, demonstrates that the greatest performance improvement is achieved by employing dual memory bank, which leverages representations generated by past query and passage encoders.
5.3 Memory bank size analysis
inniiatsssss the experimental results on the NQ dataset, demonstrating the impact of memory bank size Nimemory and accumulation steps K on CONTACCUM'S performance low-resource setting with local batch size of 8. As the memory bank size Nmemory increasee,,, more negative passages are utilized in training, and as the accumulation steps increase, more data considered in each model update. The performance of DPR in both low-resource and high-resource scenarios(NNou.
7

---

0.80
Top 20 Accuracy
0.78
DPRIV
GracCall
Preeetch ( Aors
GracAccum
Profitth W m0MM
0.76
GradAccum
GradAccum
PreBitoon Aerey 10Winal
0.74
GradAccum
Nanenary Natal
Time
Contacoum (eseeey
Conekcoum Wizel
Conteccum (enercy 5PNom))
0.72
Nemerosy Mooes
DPR DP
Ninemary -1rr
0.70
32
64
128
Total Batch Size (Ntota))
128
512
Total Batch Size (Nute)
Figure 3: Analysis of accumulation step and memory bank size. DPR performance in low-resource (BSZ=8) and high-resource (BSZ=128) settings shown as baselines, along with the performance of gradient accumulation for each total batch size (Niota)).
Figure 4: Comparison of the speed of one weight update for different methods as the total batch size (Nona) changes.
32, 64, 128) is also included for comparison. Note that gradient accumulation is not used when the total batch size and only the dual memory bank is employed.
CONTACCUM consistently outperforms GradAccum and DPR regardless of the size of memory bank and accumulation step. The results show that increasing the memory bank size improves performance even when GradAccum is not used. This indicates that even without gradient accumulation, utilizing representations from the memory bank to construct larger similarity matrix enhances performance. This trend remains consistent as the accumulation step increases. Moreover, CONTACCUM consistently outperforms GradAccum in all Now settings. Remarkably. CONTACCUM with Mical 8, Noul 64, and Vmmemoy 128 surpasses the performance of DPR in high-resource setting (Notta = Nical = 128). The performance improvement of CONTACCUM converges as the accumulation step and memory bank size increase, demonstrating that CONTAcCUM robustly enhance performance regardless of memory bank size and accumulation steps.
5.4 Train speed
In this subsection, we compare the training speed of CONTACCUM with baseline methods. Figure shows the results of experiments comparing the speed of a single training iteration (1 weight update) as the accumulation step increases low-resource settings with IIGB of available memory. Unlike the high-resource setting, where the total batch can be processed through forward and backward pass at once, the train speed slow down in low-resource settings due to various computations and storing gradients.
CONTACCUM achieves faster iteration times than GradCache, even with large memory banks. As shown in Figure CONTACCUM performs single iterations faster than GradCache in all total batch size. Notably, when Nevua 512, GradCache 93% slower than GradAccum, while CONTACCUM only takes 26% more time, even with the largest memory bank size of N Nmemory 8192. This indicates that CONTACCUM completes iterations 34% faster than GradCache. The significant additional time for computing one iteration in GradCache is due to the overhead of calculating and storing gradients of representations, as well as the repetitive forward and backpropagation. In contrast, CONTACCUM incurs relatively minor loss of speed compared to GradAccum due the additional computations involved in storing and retrieving representations from the memory bank and calculating the enlarged similarity matrix. While pre-batch negatives [19] shows similar computational efficiency to method, it degrades the performance as demonstrated in Table 2
5.5 Gradient norm ratio
conducted experiments comparing the gradient norms of query and passage encoders to investigate whether the presence of query memory bank M, affects the grudient norm imbalance problem, as discussed Section 3.3 The results are presented in Figure This experiment defines
the ratio of gradient norms between the two encoders as GradNormRatio V |/ We

---

30
(35Z=8)
WA
DPR (852-128)
ContAccum M₃)
ContAccum
Grod Nomm
15
gradhe
erenders 11P ([9ella)
0
10
15
20 Epoch
25
30
40
15
20
25
30
35
40
Epoch
Figure 3: Analysis of GradNormRatio throughout the training process on the NQ dataset.
the two encoders have similar gradient norms during training, GradNormRatio should be close to the passage encoder (gA) has larger gradient norm, GradNormRatio will be greater than 1.
measured GradNormRatio during the training of the NQ II N
Dual memory bank helps maintain gradient norm balance. The experimental results show that when the query memory bank M2 is not used, GradNormRatio consistently increases. In contrast, CONTACCUM, which utilizes a dual memory bank (Mg. Mp), maintains a GradNormRatio close to 1, similar to DPR.
This indicates that the pre- pre-batch negatives exhibit gradient norm imbalance problem. It is because pre-batch negatives only use passage memory bank, leading to an imbalance in the number of query and passage representations used in gradient calculations, discussed in 3.3 The gradient norm imbalance problem consistently occurred even when the timing of omitting the query memory bank Mg is varied during training, shown in Figure 6
The gradient norm imbalance problem observed during the actual training process becomes increasingly severe, causing the gradient norm of the passage encoder to be up 30 times larger than the query encoder. As noted by Senushkin et [33 and Chen et al. [4] such extreme differences in gradient norms between the two models negatively impact performance. The significant performance drop observed in when the query memory bank My not used can be attributed to the grudient norm imbalance problem.
6 Conclusion
In this work, we proposed CONTACCUM, novel memory reduction methodology for training dual-encoders with InfoNCE Loss in low-resource settings. By employing dual memory bank structure, CONTACCUM achieves stable training and outperforms high-resource baselines, as demonstrated through extensive experiments on five information retrieval datasets. Our mathematical analysis of the dual-encoder training process underscores the importance of balanced gradient norms, which is effectively addressed by the dual memory bank approach. Furthermore, various ablation experiments showed that the accumulation step and memory bank size significantly contribute to performance improvement.
Limitations. While CONTACCUM reduces computational costs and stabilizes training, this study is limited by its focus on supervised fine-tuning. Recently, many studies have proposed pre-training stage for dense retriever 12 29] remains to be investigated whether the gradient norm imbalance problem arises during the pre-training stage and whether CONTACCUM can alleviate it. Additionally, CONTACCUM still relies on the softmax operation, which incurs high computational costs. Reducing this reliance on the softmax operation could lead to more efficient training and broader application of the dense retriever.
Broader impacts. CONTACCUM is designed to train dense retrievers efficiently, which allows it be applied various knowledge-intensive systems with limited resources. Examples of such applications include search engines, sttrival--aumeetee machines. However, we strongly discourage the use of CONTACCUM in high-risk domains such as medical and legal fields, where the retrieval of incorrect information could have serious impact.
generation, and fact verification on local
'The values of gradient norms recorded after gradient clipping.

---

Future works. In future work, we plan to extend CONTACCUM to the pre-training phase with uni-encoder structure to assess its broader applicability. We also aim investigate efficient training strategies to mitigate the substantial computational burden caused by the softmax operation. By addressing these areas, we hope to encourage further research on optimizing dual-encoder training for low-resource settings in the field of information retrieval.
Acknowledgements
This work was supported by the National Research Foundation of Korca(NRF) grant funded by the Korea (SS/
This work also supported by Institute of Information Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) RRS-2024-00460011 Climate and Environmental Data Platform for Enhancing Climate Technology Capabilities in the Anthropocene (CEDP).
We would like express our sincere gratitude Keonwoo Kim, Joonwon Jang, Hyowon Cho, Minjin Jeon, and Sangyeop Kim for their valuable feedback and insightful comments. We also deeply appreciate our collegues; Joonghoon Kim, Sacran Park. SangMin Lee, Jiyoon Lee, Jaewon Cheon, and Seonghee Hong for their constructive discussions and support throughout this work.
10

---

References
[1] Petr Baudiš and Jan Sedivy. 2013. Modeling of the Question Answering Task in the YodaQA System. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 222-228, Cham. Springer International Publishing.
[2] Jonathan Berant, Andrew K. Chou, Roy Frostig, and Percy Liang. 2013. [Semantic Parsing on Freebase from Question-Answer Pairs In Conference on Empirical Methods in Natural Language Processing.
[3] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.04297
[4] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. 2018. Gorm Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 794-803. PMLR.
[5] Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023, Promptagator: Few-shot Dense Retrieval From Examples In The Eleventh International Conference on Learning Representations.
[6] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
[7] Luyu Gao and Jamie Callan. 2021. Condenser: a Pre-training trrintriiing Architecture for Dense Retrieval In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 981-993, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
[8] Luyu Gao and Jamie Callan. 2022. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2-53,, Dublin, Ireland. Association for Computational Linguistics.
[9] Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup. In Proceedings of the 6th Workshop on Representation Learning for NLP.
[10] Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy. PMLR.
[11] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9726-9736 Los Alamitos, CA, USA. IEEE Computer Society.
[12] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. [Ussupervised Dense Information Retrieval with Contrastive Learning
Transactions on Machine Learning Research.
[13] Bowen Jin, Hansi Zeng, Guoyin Wang, Xiusi Chen, Tianxin Wei, Ruirui Li, Zhengyang Wang. Zheng Li, Yang Li, Hanqing Lu, Suhang Wang, Jiawei Han, and Xianfeng Tang. 2023. Language Models As Semantie Indexers)
[14] Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-Scale Similarity Search with GPUs, IEEE Transactions on Big Data, 733):535-547.

---

[15] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computational Linguistics.
[16] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Doman Question Answering In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Online. Association for Computational Linguisties.
[17] Brian Kulis. 2013. Metric learning: A survey, Foundations and Trends in Machine Learning. 5(4):287-364.
[18) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics.
[19] Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning Dense Representations of Phrases at Scale. In Association for Computational Linguistics (ACL).
[20] Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent Retrieval for Weakly
SSupervised Open Domain Question Answering In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
[21] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: Bootstrapping Language-Iman Pre-training for Unified Vision-Languee Understanding and Generation In International Conference Machine Learning.
[22] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation In Advances in Neural Information Processing Systems, volume 34, pages 9705-9705. Curran Associates, Inc
[23] Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization In International Conference Learning Representations.
[24] Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie- Yan Liu, and Arnold Overwijk. 2021. Less is More: Pretrain Strong Siamese Encoder For Dense Text Retrieval Using Weak Decoder In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 2780-2791, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
[25] Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2023. Nonnaaaaeerri Masked Language Modeling Findings of the Association for Computational Linguistics: ACI. 2023, pages 2097-2118, Toronto, Canada. Association for Computational Linguistics.
[26] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyToocch: An Imperative Style, High-Performance Deep Learning Library
Advances in Neural Information Processing Systems 32, pages 1924-8035 Curran Associates, Inc
12

---

[28] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach Dense Passage |etriieal for Open-Domain Question Answering In I Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847, Online. Association for Computational Linguisties.
[29]
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to Retrieve Passages without Supervision In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2687-2700 Seattle, United States. Association for Computational Linguistics.
[30] Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence Embeddings using Siamese BERT-Networks In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
[31] Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji- Ji-Rong Wen. 2021. PAIR: Leveraging Eessage-Centrre Similarity Relation for Improving Dense Passage Retrieval In Findings of the Association for Computational Linguistics: ACL/JCCNP 2021, Online Event, August 1-6, 2021, volume ACL/IUCNPP 2021 of Findings of ACI., pages 2173-218. Association for Computational Linguistics.
[32] Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond Foundations and Trends in Information Retrieval, 3:333-389.
[33] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. 2023. Independent Component Alignment for Multi-Task Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20083-20093.
[34] Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren. 2023. Learning to Tokenize for Generative Retrieval In Advances in Neural Information Processing Systems, volume 36, pages Curran Associates, Inc.
46345 46361.
[35] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume
[36] Aäioo van den Oord, Yazhe Li. and Oriol Vinyals. 2018. Contrastive Predictive Coding CoRR, abs/ abs/1807. 0tt//807.00777
Representation Learning with
[37] Jinpeng Wang, Jieming Zhu, and Xiuqiang He. 2021. Cross-Batch Negative Sampling for Training Two-Tower Recommenders) In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 1632-1636. New York, NY, USA. Association for Computing Machinery.
[38] Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. 2020. Cross-Batch Memory for Embedding Learning. In CVPR.
[39] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. 2018. Unsupervised Feature Learning via Non- parametric Instance Discrimination In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3733-3742. Los Alamitos, CA, USA. IEEE Computer Society.
[40] Zhirong Wu, Alexei A. Efros, and Stella X. Yu. 2018. Improving Generalization via Scalable Neighborhood Component Analysis. In Proceedings of the European Conference on Computer Vision (ECCV).
[41] Lee Xiong, Chenyan Xiong, Ye Li, K Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval In International Conference Learning Representations.
13

---

[42] Zhen Yang. Zhou Shao, Yuxiao Dong, and Jie Tang. 2024. Triaampler: A Better Negative Sampling Principle for Dense Retrieall, Proceedings of the AAAI Conference on Artificial Intelligence, 38(8):926-99277
[43] Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense Text Retrieval Based on Pretrained Language Models: A Survey ACM Trans. Inf. Syst., 42(4).
14

---

A. Derivatives of InfoLoss with memory bank
aC(Sk)
Ōqi d0
VeC(Sk)
E
dqu
Neat N nenory
OC(Sk)
exp(q: D..)
Σ
log
8qu
dq.
Niacal + Ng
Amemory
Nuaa N mo
exp( DD5)
Nust N menooy
Nusa MisitNNp. ierroyy
①
i
Σ
(q P..
log
exp(q: D)))
Mocal + No memory
dqi
dqu
Nust No
Genory Nival + NP
exp(qi DD,))
Σ
Σ
da. (q: p3)) (p.pp))
Mocal + Ng Mmemury
P
Σ
Nuca Niaan explq: pld
k
Nusi + Nu
memory
Nuval N
P
Σ
Σ
Sk((.))
(q: P3
Mocal + Nu memoer
dqu
Nost + N nemorr
Pi
Σ
(Saa.)) Psl
Niccal + No Memory
al(St)
0pr
VaL(S&) Σ
Öp:
OA
PrinPt
Nueat No9 henny
OL(Sk) i
exp(q: p.
log
Öp:
Öp:
Niccal + No
memory
N Noct Natmory exp(q: qi p..)
Nucal + No
Nuxal Nasnons
i log
Σ
(q. p..
Σ
exp(q: P.)
Nincal
Ng
memory
Öp:
Öp:
Nuaa No
Nuoat Mensoyy
explq: P...
Σ
Σ
Novel Ny memory
qi
Σ
Nocat Ninuua exp(q: P
Ōp:(q: p))
Noval + No
Nusa Ny
Novcal
N7 memoey
qi
Σ
Σ
Sp(i.))
Öpi (q: p.
Nuscl + Misnny
Nocal
No
memory
q1
Σ
Su(Lt) qi
B. Details on hyperparameters
Hyperparameters. The hyperparameters for training were set as follows: the warmup step was 1,237 steps, weight decay was set 0, and a customized scheduler with linear decay the learning rate after the warmup was used. The optimizer was AdamW (23 with epsilon set to 1e-8, and the learning rate was 2e-5. Gradient clipping was applied value of 2.0, and was set 1. For retrieval, we used the FAISS [14 library to perform exact nearest neighbor search with default hyperparameters.
C. Similarity Mass
To verify whether representations generated by past encoders aid the current encoder's training, we conducted an experiment measuring the similarity mass of passage representations at different time
15

---

Figure 6: Experiments similarity probability mass.
BR0382-1281
Probabilty Mass
k000
40077
steps. The results shown in Figure 6 after passing through softmax function for all current time queries with passage representations generated at past time steps k, shown in Equation 10
The similarity mass is defined the of similarities
Q
PP--*
SimMass &
Σ j=1
Q (P)-A)
(10)
Q
Passage representations the current and previous encoder have similar importance as negative passage. The results indicate that there is no significant difference in the similarity mass between the in-batch negative passage representations at the current training step and the passage representations from to six previous steps. As shown in Equation 8 and g the gradients of the two encoders are proportional to the magnitude of the similarities. This means that negative passages with high similarity to single query produce large gradients, which aids in training the dense retrieval model [41] This finding suggests that past representations can beneficial from the early stages of training, contrary to previous studies [37 33).
Additionally, as illustrated in Figure CONTACCUM demonstrates the same similarity mass trend as DPR, validating the effectiveness of utilizing past representations from the early stages of training with CONTACCUM.
D. Gradient Norm Ratio of Omitting the Query Memory Bank
Figure 7: Experimental results of omitting query memory bank during training.
DPR (BSZ=8)
(BSZ=128)
Co 101tAccum
M
25
ContArcum (wdb Mg)
Crad Nerm Railo
ContAccum twll Mg. Epoch>101
20
ContAccum twio Mg. Epach>200
ContAccum twio Me, 30)
15
10
0-
40
Gradient norm imbalance problem occurs when the query memory bank is omitted. We omitted the query memory bank during training at various epochs: [10, 20, 30). As shown in Figure 7 the gradient norm imbalance problem arises immediately after the query memory bank is excluded. Additionally. irrespective of when the query memory bank is omitted, all experiments without the query memory bank exhibit very high gradient norm ratios in the later stages of training. This indicates that gradient norm imbalance problem can cause unstable training during the entire training process, unlike previous studies which mentioned the major cause of unstable training rapid changes in encoder representations in the carly epochs [3 37].
16

---

E. Actual Memory Usage
ContAccum uses few memory for dual memory bank but it works greatly Theoretically, CONTACCUM's query and passage memory banks( Mp) do not cache activation values, requiring only additional memory for the stored representations compared GradAccum. The memory usage of the dual memory bank can be calculated as follows:
memory X dimemhed X 2 x
(11)
where represents the query and passage memory bank and denotes full precision (4 bytes).
Table 3: Comparison of Memory Usage
Additional Memory
Method
Noccal K K/Ntotal N memory
Memory (GB)
Actual
Theoretical
DPR
8/ 1/128/
0
7.483
GradCache
8/16/128/
0
5.158
GradAccum
8/16/128/
0
8.340
CONTÁCCUM
8/16/128/
128
8.342
0.002
0.0007
CONTACCUM
8/16/128/ 512
8.346
0.006
0.0029
CONTACCUM
8/16/128/102 024
8.353
0.013
0.0059
CONTACCUM
8/16/128/5096
8.382
0.042
0.0117
Moreover, we measured each method's actual memory usage in a VRAM=11GB environment and the results are reported in table The results show that CONTACCUM uses only up to 0.5% more memory than GradAccum, which is maximum of 12MB even in the largest memory bank size(N memory consumes very limited additional memory compared GradAccum. Furthermore, while GradCache uses less memory than DPR by decomposing complex forward and backward processes, has the limitation of very slow training speed, as shown in Figure
memoory 5096). This demonstrates that CONTACCUM is memory-efficient method that
F. License
The licenses for the assets used in this paper are follows:
Overall train code and partial evaluation code from nano-DPR: CC-BY-NC 4.0
Train and evaluation datasets preprocessed by DPR: CC-BY-NC 4.0
Partial evaluation code, and train and evaluation dataset preprocessed by Beir: Apache-2.0
Hard negative generated by sentence-transformer library: Apache-2.0
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims this paper accurately represent its contributions and scope. The abstract and introduction clearly outline the proposed methods and expected outcomes, which are consistently supported by the results and discussions presented the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of the work in the Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: This paper provide the full set of assumptions and a complete proof in Section
3.1 for each theoretical result provided in Section 5
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-refereneed.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results. Specifically, the datasets and evaluation metrics are provided in Section and the details of the proposed algorithm (CONTACCUM) are given in Section 3.1 Additionally, information about the hyperparameters for the training dynamics (e.g.. optimizer, learning rate, scheduler, etc.) is provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
19

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All codes and links to download the datasets are included in the supplemental material. Additionally, we plan release the codes for reproducibility of the main experimental results after the review process to preserve anonymity.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results, See the NeurIPS code and data submission guidelines (https
//iips.c/ppul wttpccccpppccccceecoe. httpe//guuuddeeeeeuuubiiiisssss Cwt////ooeeSmssseeeeoeee
soioooooiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specify all the training and test details in the Section aad Appendix 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high computation cost of the experiments, this paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Answer: [Ye]]
Justification; The entire set of experiments was conducted using the same computer resources described in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/ http:////waappp.cpoo//a///ccttta
Answer: [Yes]
Justification:
The research conducted in this paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consideration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential Positive and negative socictal impacts are discussed in Section 6
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper only proposes efficient training algorithm for information retrieval, so it poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators original owners of assets clarified in Section 4 Also, the license of each assets are mentioned in Appendix 6
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
22

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released. the license, copyright information, and terms of use in the batasets..paersswt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "G25fKWx2kp", "reviewer": "Reviewer_jj9d", "summary": "This work proposes ContAccum, a novel approach to address the challenges of training dense retrievers within limited memory resources. This method employs a dual memory bank structure to leverage previously generated query and passage representations. It allows for the reuse of past representations without the need for extensive computational resources, thus making the training process both faster and more robust. Through experiments conducted on a variety of information retrieval datasets, the paper demonstrates that ContAccum outperforms not only current memory reduction methods but also scenarios with high resource availability. This indicates that the method is highly effective even under resource-constrained environments.", "questions": "* In Figure.1, the meaning of the blue and red colors are not clear, and the Contrastive Accumulation mechanism in Figure1.(c) is hard to understand. For example, what is negative samples there? how can we read the numbers in the captions?\n\n* In Table.1, DPR and GradAccum display similar performance under the *VRAM=24GB* setting; however, under the *VRAM=11GB* condition. Can we gain any insightful interpretations from these results?\n\n* Line 102, I don't quite understand \"multiple high-cost accelerators here\".", "limitations": "The authors have claimed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Training dense retrievers utilizing the InfoNCE loss function requires a large batch size, which can pose challenges under GPU memory constraints. ContAccum is an novel solution, specifically tailored to alleviate this issue by optimizing the training process to be more memory-efficient, thereby enabling effective learning even in environments with limited GPU capacity.\n* Experimental evaluations across various benchmark datasets have demonstrated that ContAccum outperforms competing methods such as GradAccum and GradCache, well showcasing its efficacy.\n* The ablation studies validate the role of each constituent element in the proposed method. Specifically, these studies underscored the indispensable nature of components like the query memory bank ($M_q$), the utilization of past encoder representations, and the GradAccum technique.", "weaknesses": "* Lee et al. [1] introduced a passage memory bank ($M_p$) for training dense retrievers, and while ContAccum extends this with an added query memory bank ($M_q$), the perceived novelty may be somewhat overshadowed by Lee's precedent.\n* From my perspective, the method appears akin to an asymmetric sampling technique for queries and passages within datasets, implementing backpropagation selectively with a portion of these elements. While this approach may facilitate training under certain constraints, it arguably does not introduce significant innovation to the repertoire of effective training algorithms prevalent in the machine learning community.\n* Figure.1 and Figure.2 are quite confusing, and I can not read out how the methods work.\n\n**minor issues**\n\n* In Line 214-223, many spaces are missed in the paragraph.\n* Line142, $\\nabla\\Lambda$ should be $\\nabla_\\Lambda$ .\n* In equation.(5), what's the meaning of $sg(\\cdot)$? I doubt it is stop-gradient but I didn't find it in the contexts. \n\n[1]  Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning Dense Representations of Phrases at Scale. In *Association for Computational Linguistics (ACL)*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weaknesses\n**Novelty of ContAccum compared to pre-batch negatives**\nContAccum directly addresses the primary limitation of PreBatch: its inherent instability during the training process. PreBatch did not consistently enhance performance throughout the training epochs, a problem highlighted in both the original PreBatch work and our Table 2 experiments ($w/o M_q$), where PreBatch was applied from the first epoch. This significant instability prevents PreBatch from effectively approximating high-resource performance under memory-constrained conditions, thus limiting its practical utility in real-world scenarios.\n\nAs stated in our global rebuttal, we theoretically and experimentally identified the causes of PreBatch instability as gradient imbalance problem and proposed a dual memory bank structure to address these issues. As a result, unlike PreBatch, ContAccum, which leverages GradAccum and a dual memory bank, is a versatile method applicable to various dense retriever models [1, 2].\n\n**Impact for machine learning community**  \nContAccum uses a dual memory bank to overcome the limitations of GradAccum, such as the small number of negative samples, and stabilizes training under memory constraints. We focused on improving performance by tightening the lower bound of mutual information determined by the number of negative samples [3].\n\nWe agree that our method resembles asymmetric sampling techniques with the use of various forms of representation. However, our approach differs from traditional asymmetric sampling methods, which aim to improve generalization performance through multi-view generation [4] or data augmentation for imbalance class distribution [5]. In contrast, we propose a dual memory bank structure to achieve low computational and memory overhead and stable training under memory constraints.\n\nMoreover, given the importance of retrievers in the RAG framework and the high memory demands of large batch sizes during dense retriever training, our method enables efficient training of dense retrievers under real-world memory constraints. ContAccum’s use of InfoNCE Loss also suggests potential for future research extensions beyond dense retrievers to vision and vision-language representation learning, where large memory requirements are common.\n\nAdditionally, if you could provide more specifics regarding the asymmetric sampling methods you mentioned, we believe it would facilitate a more detailed discussion on this topic.\n\n# Questions\n**Clarity of Figures 1 and 2**  \nFigure 1 compares existing memory reduction methods GradAccum and GradCache with our proposed method, ContAccum. The elements depicted in the figure are as follows:\n- Blue: query representation\n- Red: passage representation\n- Gray: representations cached in the memory bank, indicating the total memory bank size ($N_\\text{memory}$) where no gradients occur\n- Rectangle with rounded corners (surrounding circles): local batch ($N_\\text{local}$) of query and passage representations\n- Positive pair: i$^th$ query and passage representation, with the positive pair’s similarity score represented as the diagonal element in the similarity matrix\n\nIn this context, the number of negative samples increases in the order of ContAccum > GradCache > GradAccum, as indicated by the size of the similarity matrix in Figure 1.\n\nFigure 2 provides a detailed description of ContAccum across accumulation steps. ContAccum employs a dual memory bank alongside gradient accumulation, constructing a larger similarity matrix with additional representations from memory bank at each accumulation step to calculate the InfoNCE Loss. Furthermore, at each accumulation step, the oldest representations in the memory bank are replaced with the newly generated ones. This process is illustrated in Figure 2, where in accumulation step 1 ($k=1$), the oldest representations ($P^{t-1}_1, Q^{t-1}_1$) are dequeued and the newly generated representations ($P^t_1, Q^t_1$) are enqueued.\n\nA more detailed explanation can be found in Section 3.1 for contrastive learning with naive gradient accumulation, and Section 3.2 for ContAccum using a memory bank. \n\n**Insights on VRAM Setting Performance**  \nWhen training DPR with GradAccum, the number of negative samples remains the same as with low-resource setting, but the number of samples used in gradient computations increases. Therefore, the performance gain of GradAccum compared to DPR is attributed to the stable gradient generation achieved through larger batch size, a common trait in deep learning methodologies.\n\nIn table 1, DPR and GradAccum show similar performance with VRAM=24GB, but not with VRAM=11GB. This is because, under VRAM=11GB, despite having the same number of negative samples, 7, DPR computes relatively unstable gradients with only 8 samples, whereas GradAccum achieves stable gradients with 128 samples. In contrast, with VRAM=24GB, DPR generates sufficiently stable gradients with 32 samples, resulting in similar performance to GradAccum with 128 samples.\n\nHowever, in both scenarios, our proposed ContAccum outperforms GradAccum . As mentioned in the global rebuttal, this is due to ContAccum’s ability to utilize more negative samples than DPR, GradAccum, and GradCache in low-resource settings.\n\n**Clarity of “multiple high-cost accelerators”**  \nHigh-cost accelerators refer to advanced GPUs or similar devices that provide substantial memory but come at a high price.\n\n[1] Shuqi Lu et al. Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder.  \n[2] Lee Xiong et al. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval.  \n[3] Oord, Aäron van den et al. Representation Learning with Contrastive Predictive Coding.  \n[4] Chengchao Shen et al. Asymmetric Patch Sampling for Contrastive Learning.  \n[5] Haotao Wang et al. Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yIULJkdMQ", "reviewer": "Reviewer_sDHF", "summary": "The paper begins by addressing the weaknesses of traditional methods, specifically pointing out that Gradient Accumulation utilizes fewer negative passages and Gradient Cache requires extensive training time. To overcome these issues, the authors propose a novel method called Contrastive Accumulation. This method builds upon the gradient accumulation approach by adding two memory banks to store previous encoder information for both queries and passages. This enhancement allows for the efficient use of previous information when calculating gradients, ensuring the utilization of sufficient negative passages and maintaining computational efficiency. Additionally, the method controls the size of the memory bank to ensure a stable training process.", "questions": "* For the performance comparison between the pre-batch negative method and COINTACCUM, what batch size was used? Additionally, in Figure 3, what is the performance of DPR when the batch sizes are 32 and 64?\n* Regarding Figure 4, I am curious about the training speed of the pre-batch negative method. Since you added a memory bank compared to the pre-batch negative method, would this significantly affect the training speed?\n\nPlease also refer to the weakness section.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-organized and clearly written.\n* The comments and explanations are clear, concise, and follow a coherent narrative.\n* The experiments conducted validate the authors' claims, demonstrating that their method achieves higher scores in most cases under low-resource settings. Furthermore, it is more time-efficient than GradCache while maintaining comparable performance.\n* The analysis of the instability in training and the proposed solution to balance the gradients of the two encoders to ensure stable training is particularly interesting.", "weaknesses": "* It is good to illustrate the pros and cons of GradAccum and the motivation behind proposing CONTACCUM, it would benefit from a detailed analysis of how ContAccum overcomes the limitations of GradCache.\n* The paper only presents the scores of DPR in high-resource settings. A comparative experiment of GradAccum, GradCache, and ContAccum in high-resource settings would be beneficial to demonstrate that CONTACCUM's performance is not inferior to the other two traditional methods.\n* The introduction of DPR is missing. Since DPR is widely referenced in the experimental section, a related introduction at the beginning would be helpful.\n* The paper contains numerous typos and formatting mistakes.\n\nTypo\n* Some typo exist in line 120, the “GradAccumcannot” should change to “GradAccum cannot”\n* At line 55, it seem to use the wrong word, reducing the training “speed”, or you want to use “time”.\n* Line 207,208 “CONTACCUMexhibits” should change to “CONTACCUM exhibits”\n* Line 210 “GradAccumimproves” should change to “GradAccum improves”\n* Line 211 “CONTACCUMare” should change to “CONTACCUM are”\n* Line 283 “GradNormRatioduring” to “GradNormRatio during”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n===\n**Detailed Analysis Compared to GradAccum and GradCache**  \nAs explained in the global rebuttal,  ContAccum achieves similar training speed to GradAccum and faster than GradCache while outperforming both in terms of performance. This is because ContAccum utilizes simple forward/backward operations similar to GradAccum and uses a memory bank to include more negative samples. Moreover, as ContAccum's memory bank size increases, it can use more negative samples than high-resource settings, enabling it to achieve higher performance than high-resource DPR despite being a memory reduction method (Table 1).\n\nThe key differences between ContAccum and GradCache are in 1) _the process of the forward and backward pass_ and 2) _the number of negative samples used._ GradCache approximates a high-resource setting by first calculating the loss without saving activations and then performing backward propagation to each representation. It subsequently splits the total batch into local batches, forward each local batch again while saving activations, and finally computes the gradient for each local batch. Despite this complex process, GradCache uses the same number of negative samples as the total batch.\n\nIn contrast, ContAccum forwards only the each local batch but caches more negative samples than the total batch size. This simpler and more efficient approach results in superior performance gains for ContAccum.   \n**Additional Experiments Under High-Resource Settings**  \nWe thank the suggestion for additional experiments, which will enhance our paper. The purpose of Table 1 was to compare the performance of various methods in low-resource settings against high-resource DPR, which served as a reference performance. The objective of memory reduction methods is to approximate high-resource performance in low-resource conditions.  \nBased on the comment, we conducted additional experiments on the MS Marco, a widely utilized benchmark dataset, to provide further insights within the limited time of the rebuttal period. The results are presented in Table 1 of the attached PDF. From these new experiments, we derived two key insights:  \n- _Total Batch Size Matters_: All methods show significantly improved performance compared to previous low-resource experiments. This improvement is attributed to the increased total batch size (512), which leads to more stable gradients during model updates.\n- _ContAccum Outperforms in High-Resource Settings:_ Even in high-resource scenarios, ContAccum demonstrates higher performance compared to other methods. This indicates that ContAccum consistently shows robust performance regardless of resource size.\n\n**Introduction of DPR**  \nWhile Sections 1 (lines 18-20) and 3.1 (lines 94-100) provide a brief explanation of DPR, we understand that a more detailed description is needed. We will enhance the explanation of DPR in camera ready.\n\nQuestions\n===\n**Batch Size and Performance in Table 2 and Figure 3**  \nThe experiments in Table 2 were conducted with the same settings as in Table 1 (VRAM=11GB, Total Batch size=128, Local Batch size=8, Accumulation Step=16). Figure 3 experiments were designed to check DPR performance approximation in low-resource situations ($N_\\text{local} = 8$) compared to high-resource situations ($N_\\text{total}=128$).  \nWe've conducted additional experiments, the results of which are shown in Figure 3 (PDF). ContAccum, GradAccum, and DPR ($N_\\text{total} = 8$) were run with VRAM=11GB, while DPR ($N_\\text{total}=32, 64, 128$) used VRAM=80GB. We learned two key findings.  \n- _ContAccum consistently outperforms high-resource DPR:_ Despite DPR (BSZ=32, 64, 128) being trained with larger VRAM, ContAccum achieves higher performance regardless of total batch size. This confirms that ContAccum can outperform various high-resource scenarios despite being a memory-reduction method.\n- _Larger memory banks boost performance:_ Performance improves as the memory bank ($N_\\text{memory}$) size increases, regardless of total batch size ($N_\\text{total}$). Notably, when $N_\\text{memory} > 4*N_\\text{total}$, ContAccum outperforms all high-resource DPR scenarios (BSZ=32, 64, 128), likely due to the utilization of more negative samples.\n\n**Training Speed of Pre-Batch Negatives**  \nComparing the training speed of pre-batch negatives, which only use the passage memory bank ($M_p$) in ContAccum, offers a clear evaluation. We thank you for this insightful suggestion.  \nWe measured the training speed of GradAccum with pre-batch negatives. The experimental setup, including GPU, accumulation step, and memory bank size, was identical to ContAccum, except that only the passage memory bank was used. The results are included in Figure 4 (PDF). As shown, ContAccum shows minimal speed degradation compared to pre-batch negatives, with an average 2.1% speed reduction when using the same $N_\\text{memory}$.  \nDespite this minor speed difference, ContAccum maintains stable training in low-resource settings, unlike PreBatch (Table 2 and Figure 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z45JoUajdP", "reviewer": "Reviewer_ADDC", "summary": "This paper proposes a method for training with contrastive loss with more memory efficiency and stability. The paper focuses on setups with memory constraints. In such settings, the batch size cannot be too large which leads to limited negative samples for computing per-batch contrastive loss. Prior work, GradAccum, proposed to compute the gradient per small mini-batch by considering negatives within the mini-batch but then accumulating the gradient over multiple mini-batches. This results in effectively computing a block-diagonal subset of values in the full contrastive matrix between all samples in the summation.\n\nThis paper proposes ContAccum, which improves GradAccum by keeping the representations of previous keys and queries in a memory bank, while not backpropagating the gradients to previous mini-batches. This expands the positives and negatives for each mini-batch while adding a small overhead in computing the contrastive loss. In order to prevent a gradient imbalance between positives and negatives, the proposed method has a memory bank for both keys and queries instead of for negatives only.\n\nThe memory keeps the previously generated query and passages in a memory bank to be used in future contrastive losses.", "questions": "- Can the authors analyze the overhead of ContAccum in holding the memory bank compared to other methods’ overheads in terms of Bytes in each setting? I understand the batch size fits in each setup, but it helps to see this overhead as a fraction of the total memory GBs.\n\n\nTypos: A space is missing everywhere after GradAccum/GradNormRatio. If it is a macro, consider calling it with \\macro{} or use other spacing tools inside the definition of the macro.", "limitations": "The paper has discussed its limitations and broader impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method improves on DPR and GradAccum in 11/12 tasks in the sitting with 11GB memory and 9/12 tasks in the setting with 24GB memory.\n- The proposed method outperforms DPR with low memory.\n- The gap in accuracy between the proposed method and prior works is bigger at low memory settings (up to 11% with DPR at 11GBs).\n- The ablations comprehensively analyze the effect of each component and show that balancing the gradients with double memory banks is helpful.", "weaknesses": "- Is the method specific to dense retrievers? The paper is missing background on dense retrievers and it is not clear why the method is specific to dense retrievers rather than generally training with a contrastive loss. For example, could this method be used for contrastive learning of images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weakness \n**Broader Applicability**  \nContAccum has the potential to be applied to various tasks and modalities. As mentioned in the global rebuttal, while ContAccum was initially proposed for training dense retrievers, its application can be extended to other domains due to the similarities in training frameworks and tasks. These fields usually require large batch sizes, making memory reduction methods crucial for real-world applications. As the first memory reduction method using dual memory banks, ContAccum could extend beyond information retrieval to other domains.\n\nIn this paper, Dense Retrievers have unique aspects that distinguish them from other tasks and domains:\n1. _Model Structure:_ Existing studies using contrastive learning for unsupervised image representation learning [1] typically focus on single-encoder training. In contrast, the dense retriever based on DPR involves training a dual encoder for queries and passages. This architectural difference may lead to distinct gradient imbalance issues when using memory banks, requiring further analysis.\n\n2. _Asymmetric Loss:_ Due to the nature of the information retrieval task, DPR only trains with a query-to-passage InfoNCE Loss. In contrast, vision-language domain methods have a symmetric loss for image-to-text and text-to-image tasks. This difference requires additional analysis of the gradient imbalance problem.\n\n3. _Model Initialization:_ DPR initializes with pre-trained BERT, whereas methods like CLIP [2] often start with random initialization  This can result in different characteristics during the early stages of training. For valid memory banks strategy, the cached representations need to be meaningful negative samples. However, randomly initialized encoder generates unstable representation. Therefore, further research on the effectiveness of early-stage representations with random initialization is needed.\nWe believe that extending research on these three points will position ContAccum as a valuable contribution to memory reduction techniques across various tasks and modalities.\n\n# Questions\n**Actual memory usage**   \nTheoretically, ContAccum's query and passage memory banks ($M_q$, $M_p$) do not cache activation values, requiring only additional memory for the stored representations compared to GradAccum. The memory usage of the dual memory bank can be calculated as follows:\n\n$N_\\text{memory} \\times \\text{embedding dimension} \\times 2 \\times 4$  \n\nHere, \"2\" represents the query and passage memory banks ($M_q, M_p$), and \"4\" denotes full precision (4 bytes).  \nMoreover, we measured each method's actual memory usage in a VRAM=11GB environment and the results are reported in the following table. The results show that ContAccum uses only up to 0.5% more memory than GradAccum, which is a maximum of 12MB even in the largest memory bank size($N_\\text{memory}=5096$). This demonstrates that ContAccum is a memory-efficient method that consumes very limited additional memory compared to GradAccum.\n\nFurthermore, while GradCache uses less memory than DPR by decomposing complex forward/backward processes, it has the limitation of very slow training speed, as shown in Figure 4.\n\n| Method    | $N_\\text{local}/K/N_\\text{total}/N_\\text{memory}$ | memory(GB) | addional memory usage (GB(%)) | theorecal memory usage (GB) |\n|-----------|-------------------------------------------------|------------|:------------------------------:|:------------------------:|\n|    DPR    |                    8/1/128/0                    |    7.483   |                              |                        |\n| GradCache |                    8/16/128/0                   |    5.158   |                              |                        |\n| GradAccum |                    8/16/128/0                   |    8.340   |                              |                        |\n| ContAccum |                   8/16/128/128                  |    8.342   |         0.002(0.024%)        |       0.0007       |\n| ContAccum |                   8/16/128/512                  |    8.346   |         0.006(0.072%)        |       0.0029       |\n| ContAccum |                  8/16/128/1024                  |    8.353   |         0.013(0.156%)        |       0.0059       |\n| ContAccum |                  8/16/128/5096                  |    8.382   |         0.042(0.504%)        |       0.0117       |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the three anonymous reviewers for their insightful feedback. We have made our best efforts to respond to all points raised and provide clarification where needed. Furthermore, These comments have provided us with an opportunity to strengthen our statement through additional experiments. Below are our responses to common questions:

**Major Comments** 

**1. Novelty of ContAccum Compared to Baselines (GradAccum, GradCache)**  
ContAccum utilizes a Dual Memory Bank to combine the fast training speed of GradAccum's simple forward and backward computations with the performance improvements from GradCache's use of numerous negative samples. As shown in Figure 2, ContAccum stores query and passage representations generated from the previous step's forward pass This allows the use of more negative samples compared to the total batch size's negative samples . Also, the dual memory bank structure enabled more stable training compared to utilizing only the passage memory bank ($M_p$).  
Our proposed method not only achieves higher performance than existing memory reduction methods in low-resource settings (Table 1) but also demonstrates a 34% faster training speed compared to GradCache (Figure 4).  
- Performance (Table 1): ContAccum outperforms GradAccum, GradCache, and even high-resource DPR by securing more negative samples through the dual memory bank. While GradAccum uses fewer negative samples ($N_\text{local} - 1$) and both GradCache and DPR use $N_\text{total} - 1$ negative samples, ContAccum uses more negative samples ($N_\text{memory} + N_\{textlocal} - 1$) by utilizing the memory bank.  
- Training Speed (Figure 4): ContAccum requires only 26% additional time compared to GradAccum, as it utilizes representations in the memory bank for similarity matrix construction. In contrast, GradCache shows up to 93% speed reduction compared to GradAccum due to its complex forward and backward processes.

**2. Relationship with Pre-Batch Negatives (PreBatch)**  
PreBatch caches previously generated passage representations in a memory bank for calculating similarity matrix, aiming to increase negative samples. However, using PreBatch in memory reduction scenarios leads to highly unstable performance and limited performance improvement (Figure 5 and Table 2).  
Our gradient analysis in Section 3.3 demonstrates the performance instability of PreBatch as the gradient imbalance problem between query and passage encoders. Figure 5 confirms that PreBatch indeed suffers from the gradient imbalance problem, leading to performance degradation.  
ContAccum maintains a balance in gradient norm between query and passage encoders similar to DPR during the training process by utilizing both query and passage memory banks.  

In summary, unlike PreBatch, ContAccum can be widely applied as a simple and effective method for training dense retrievers by using a dual memory bank structure with GradAccum. 

**3. Additional Insights During the Rebuttal Period**  
Below are insights from additional experiments conducted during the rebuttal period. The attached PDF contains detailed results.  

- _ContAccum consistently improves performance regardless of the target high-resource total batch size._ Figure 3 in the attachment shows that ContAccum consistently achieves higher performance across various total batch sizes (BSZ=32, 64, 128) by changing accumulation steps. We also found that a memory bank size (N_memory) of 4 times the total batch size is sufficient to surpass high-resource performance.
- _ContAccum requires minimal additional time compared to PreBatch._ As shown in Figure 4 in the attachment, the training time difference between PreBatch and ContAccum is minimal when applying GradAccum. In the same experimental setup (accumulation step, memory bank size), ContAccum only required 2.1% additional time compared to PreBatch. However, ContAccum allows for stable training despite this small difference in training speed (Table 2, Figure 5 in the original submission).
- _ContAccum performs well even in high-resource settings._ As demonstrated in Table 1 in the attachment, applying ContAccum results in higher performance compared to GradAccum and GradCache. This suggests that ContAccum exhibits robust performance improvements across various resources and local batch sizes.

**4. Impact on the Machine Learning Community**

- _ContAccum is a memory reduction method applicable to dense retriever training,_ which is crucial in the information retrieval system and Retriever-Augmented Generation (RAG) Framework. Many Dense Retriever models utilize InfoNCE Loss in high-resource settings. With the growing importance of memory reduction methods in real-world applications, our proposed method can reduce training costs while improving performance.
- _ContAccum has the potential to be applied to various modalities and learning frameworks._ ContAccum utilizes InfoNCE Loss, which is widely applied in representation learning for Vision / Vision-Language domains. These fields also require significant computational resources, increasing the need for memory reduction research. [1, 2]  Given that ContAccum uses InfoNCE Loss, it has the potential to be applied to various domains and modalities beyond dense retrievers. Future research could extend its applicability to other domains and modalities. 

**Minor Comments:**
We appreciate the reviewers’ careful reading of our manuscript and their identification of typographical errors and misformattings. We will correct all the typos and misformatted words across lines 55, 120, 142, 210-224, and 283 regarding the use of each method’s name.

[1] Alec Radford et al.  Learning Transferable Visual Models From Natural Language Supervision.  
[2] Mehdi Cheriti et al. Reproducible scaling laws for contrastive language-image learning

Meta-review: This work introduces ContAccum, a novel method for training dense retrievers with limited memory by reusing previously generated query and passage representations through a dual memory bank structure. Experiments show that ContAccum outperforms existing memory reduction methods and remains highly effective even in resource-constrained settings.

Most reviewers actively participated in the discussion and raised their scores based on the rebuttal, which greatly contributed to improving the overall quality of the paper.

After rebuttal, all reviewers unanimously agreed on acceptance. I also consider this a solid paper and recommend acceptance.
Decision: Accept (poster)