SlowGuess/NAACL_2021_OCR huggingface dataset:
As described in Section 3.2, CoRT projects the context representation of the underlying encoder  $\tau$  to an arbitrary representation size  $e$ . This size determines the size of the final index and also influences the retrieval latency. The total size of the encoded corpus is easy to calculate. For example, with  $e = 128$  and the MS MARCO corpus, the index size (without overhead) amounts  $8.8M$  documents  $\times$  128 floats/document  $\times$  4bytes/float  $\approx$ $4.5 \times 10^{9}$  bytes  $\approx$ $4.2GB$ . Thus,  $e$  is proportional to the total size and reducing  $e$  to 64 would halve the memory footprint. If  $e$  is small, however, it is more difficult to attain the training objective. Thus,  $e$  can be used for a trade-off between ranking quality and computational effort / resource cost. We investigate the relation between the representation size  $e$  and the ranking quality by conducting identical training runs with different numbers for  $e$ . The results in Table 3 show that MRR@10 already saturates at  $e = 128$ . Interestingly, even with an representation size of  $e = 32$ , CoRT outperforms BM25 in terms of MRR@10 and nDCG@20 by a big margin.
Table 3: First-stage ranking results for various representation sizes.
e	MRR @10	nDCG @20	RECALL @200 @1k
CoRT	32	23.6	29.8	70.6	81.8
64	25.6	32.3	75.6	85.7
128	26.8	33.4	77.2	87.1
256	26.8	33.6	78.2	87.8
768	27.1	34.0	78.6	88.0
CoRTBM25	32	25.2	33.5	85.2	93.7
64	26.6	34.9	86.2	94.4
128	27.4	35.7	87.0	94.8
256	27.2	35.6	87.0	94.9
768	27.4	36.0	87.3	94.9
4In our definition, saturation is reached, when doubling the number of candidates results in less than  $0.5\%$  increase of the respective metric
4.5 Latency Measurements
We propose two methods for the deployment of CoRT. The first exhaustively calculates similarity scores using multiple GPUs while the second incorporates an Approximate Nearest Neighbor index (ANN). We measure retrieval latencies of those methods and compare them with BM25 as representative for term-based retrieval models based on inverted indexing. We conduct the latency measurement based on the top-1000 retrieval for the dev.small split of the MS MARCO passage corpus. Since some approaches profit from batch computing, we also measure the latency for batches of 32 queries. As representation size, we have chosen  $e = 128$ , since it is the smallest representation size investigated in Section 4.4 that does not hurt the ranking quality of CoRT $_{\text{BM25}}$ .
4.5.1 Lucene BM25 Baseline
As retrieval latency baseline, we use a Lucene index generated by the Anserini toolkit (Yang et al., 2017). Please note, this is not perfectly representative for sparse bag-of-words retrieval: Retrieval latency can be reduced due to index pruning without significantly hurting retrieval quality (Mackenzie et al., 2020). The retrieval was performed on a machine with an Intel Core i9-9900KS processor (16 logical cores, 8 physical) and enough memory to hold the whole corpus. Single queries were processed using the single-threaded search function, while batch-wise search has been performed with 16 threads.
4.5.2 Retrieval using multiple GPUs
Multiple GPUs can be used to deploy CoRT for fast large-scale ranking. We propose to uniformly distribute the vector representations of the corpus on the available GPUs. Each GPU ranks its own partition of the corpus as described in Section 3.4. Afterwards, the results for each partition are aggregated by selecting the top-k candidates with highest scores.
4.5.3 Retrieval using ANN
Since CoRT operates on vector similarities, it can make use of ANN search. We measure the retrieval latency and the loss of recall, which occurs due to the pruning heuristics. We use a graph-based index optimized with the ONNG method (Iwasaki and Miyazaki, 2018). An implementation of this method is publicly available as part of the NGT
Table 4: Retrieval latencies averaged over 6980 queries of the dev.small split.
MS MARCO
Passage (dev. small)	RECALL
@200	LATENCY (ms)
Single	Batch32
BM25 (anserini)	73.8	38	290
CoRT (e = 128)
Query Encoding
- Single GPU
Retrieval	-	8	17
- Single GPU	77.2	68	164
- Quad GPU	77.2	17	35
- ANNε = 0.01	76.6	4	-
- ANNε = 0.1	76.9	17	-
- ANNε = 0.4	77.2	71	-
CoRTBM25 Total
- Quad-GPU	87.0	~63	~342
- ANNε = 0.1	86.9	~63	-
- ANNε = 0.01	86.8	~50	-
Library<sup>5</sup>. To adjust the trade-off between retrieval latency and accuracy of the index, we alter the search range coefficient  $\epsilon$ . We always retrieve 1000 candidates from the ANN index, even if we use a smaller number of candidates in a ranking pipeline.
4.5.4 Latency Measurements
The latency measurements are reported in Table 4. For CoRT the total retrieval latency per query consists of two factors: Query encoding and retrieval. The query encoding has to be performed by the query encoder  $\psi_{\beta}$ , which we highly recommend to run on a GPU. The latency of the retrieval depends on the retrieval methods described above. Employing multiple GPUs appears to reduce retrieval latency on a linear scale: The exhaustive search using four GPUs takes  $17\mathrm{ms}$  for a single query, while a single GPU takes four times as long. The total retrieval time per query sums up to  $17 + 8 = 25\mathrm{ms}$  for the quad GPU setting, which is below the BM25 baseline. It is worth noting, that batch-wise processing results in a substantial efficiency increase: Retrieval for 32 queries at once only takes about twice as long as a single query. This can be useful if multiple queries queue up while the system is busy. The tested BM25 index (Anserini), on the other side, seems to suffer from multiprocessing overhead or some sort of bottleneck. The latencies for the ANN index has been measured with three different values for the search range coefficient  $\epsilon$ . While this significantly affects the retrieval latency, only slight differences regarding the quality of the first-stage ranking are
<sup>5</sup>https://github.com/yahoojapan/NGT
observed. Latencies for  $\mathrm{CoRT}_{\mathrm{BM25}}$  comprise latencies from BM25, CoRT's query encoding and the corresponding retrieval method. For simplicity, we assume sequential processing of all three components, although BM25 could be processed in parallel.
4.6 End-to-end Retrieval
Intrigued by the remarkable ratio of retrieval latency and ranking quality of ColBERT's full-ranking approach (Khattab and Zaharia, 2020), we used our above findings to create a competitive end-to-end ranking setup. We suggest to re-rank the top-64 candidates from  $\mathrm{CoRT}_{\mathrm{BM25}}$  with  $e = 128$ , retrieved by an ANN index ( $\epsilon = 0.1$ ). The end-to-end latency comprises 8ms for query encoding, 17ms for CoRT retrieval based on ONNG, 38ms for BM25 retrieval, and 192ms for re-ranking. Although the BM25 candidates could be retrieved in parallel, we report sequential processing latencies. As shown in Table 5, we outperform ColBERT's end-to-end ranking performance in terms of MRR@10 and retrieval latency. It is worth noting that the "RTX 2080 Ti" we used for latency measurements is less powerful than the "Tesla V100" Khattab and Zaharia (2020) used for their measurements. CoRT's representations for the MS MARCO corpus only weight 4.3GB when  $e$  is set to 128, or 7.0GB when indexed in an ONNG index. The size of the query encoder only amounts about 50MB, which is due to ALBERT's parameter sharing. To compile the full  $\mathrm{CoRT}_{\mathrm{BM25}}$  candidates, the corresponding BM25 index is needed, which amounts 2.2 GB on disk. Although more memory is needed to deploy and operate both indexes, this is by far less than the 154GB footprint reported by Khattab and Zaharia (2020) for ColBERT's end-to-end approach.
Table 5: Measured end-to-end ranking quality and latency. * ColBERT's latency was measured with different hardware
MS MARCO
Passage (dev. small)	MRR @10	LATENCY (ms)
ColBERTL2 (Khattab and Zaharia, 2020)	36.0	458*
CoRTBM25 (ANNε = 0.1, top-64) + BERT Re-ranking	38.4	255
In this paper, we propose CoRT, a framework and neural first-stage ranking model that leverages contextual representations from transformer-based language models to complement term-based ranking functions. As a result, we observe decently increased recall measures and improved end-to-end ranking quality on the MS MARCO passage task. Also, we are able to decrease the number of candidates for re-ranking without hurting the final performance. Our further experiments reveal sweet spots for CoRT's representation size and the number of re-ranking candidates. We presented two deployment strategies for CoRT and measured their performances in terms of efficiency and effectiveness. Finally, we demonstrate CoRT can be used with a simple BERT-based re-ranker to create a competitive ranking pipeline.
We would like to thank Felix Hamann and Prof. Dr. Adrian Ulges for helpful discussions and comments on the manuscript, as well as the anonymous reviewers for their valuable feedback. This work was funded by German Federal Ministry of Education and Research (Program FHprofUnt, Project DeepCA (13FH011PX6)).
Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg. 2016. Off the beaten path: Let's replace term-based retrieval with k-nn search. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 1099-1108. ACM.
Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using A "siamese" time delay neural network. Int. J. Pattern Recognit. Artif. Intell., 7(4):669-688.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, pages 169-174. Association for Computational Linguistics.
Ruey-Cheng Chen, Luke Gallagher, Roi Blanco, and J. Shane Culpepper. 2017. Efficient cost-aware cascade ranking in multi-stage retrieval. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 445-454. ACM.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820.
Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1533-1536. ACM.
Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 126-134. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Modularized transformer-based ranking framework. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4180-4190. Association for Computational Linguistics.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 55-64. ACM.
Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. 2020. A deep look into neural ranking models for information retrieval. Inf. Process. Manag., 57(6):102067.
Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Neural vector spaces for unsupervised information retrieval. ACM Trans. Inf. Syst., 36(4):38:1-38:25.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, San Francisco, CA, USA, October 27 - November 1, 2013, pages 2333-2338. ACM.
Masajiro Iwasaki and Daisuke Miyazaki. 2018. Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data. CoRR, abs/1810.07355.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA. Association for Computing Machinery.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Efficient document re-ranking for transformers by precomputing term representations. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 49-58. ACM.
Joel M. Mackenzie, Zhuyun Dai, Luke Gallagher, and Jamie Callan. 2020. Efficiency implications of term weighting for passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1821-1824. ACM.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 collocated with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR, abs/1901.04085.
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to doctttttquery. Online preprint.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. CoRR, abs/1904.08375.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the behaviors of BERT in ranking. CoRR, abs/1904.07531.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. Learning semantic representations using convolutional neural networks for web search. In 23rd International World Wide Web Conference, WWW '14, Seoul, Republic of Korea, April 7-11, 2014, Companion Volume, pages 373-374. ACM.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted
structured feature embedding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4004-4012. IEEE Computer Society.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Lidan Wang, Jimmy J. Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011, Beijing, China, July 25-29, 2011, pages 105-114. ACM.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 55-64. ACM.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1253-1256. ACM.
Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik G. Learned-Miller, and Jaap Kamps. 2018. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 497-506. ACM.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jababaie. 2020. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A Appendix: Implementation Details
A.1 Hardware & Software
We use up to four NVIDIA GTX 2080 TI graphic cards in combination with 128GB DDR4 RAM and an Intel Core i9-9900KS processor. We use PyTorch (Paszke et al., 2019) and HuggingFace's Transformers (Wolf et al., 2019) as deep learning libraries. BM25 rankings are generated using the Anserini toolkit (Yang et al., 2017).
A.2 CoRT Training
We train CoRT based on the pretrained ALBERT model "albert-base-v2", which is the lightest available version in HuggingFace's repository. Each model is trained for 10 epochs, where each epoch includes all queries that are associated to at least one relevant document plus one randomly sampled positive and one negative passage. Negative examples are sampled from unlabeled passages of top-100 BM25 rankings. There, we exclude the first 8 ranks to reduce the probability of drawing actual relevant passages and thus give contradictory signals less often. We find this slightly increases CoRT's ranking quality. As usual for BERT-based models we use the ADAM optimizer with weight decay fix (Loshchilov and Hutter, 2019) and the default parameters  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $eps = 10^{-6}$ , a weight decay rate of  $\lambda = 0.1$  and a linearly decreasing learning rate schedule starting with  $lr = 2 \times 10^{-5}$  after
2.000 warm-up steps. We train mini-batches of size  $n = 6$  samples (triples) while accumulating the gradients of 100 mini-batches before performing one update step. The triplet margin is set to  $\mu = 0.1$ , which has been tuned within the range of [0.01, 0.2].
A.3 Re-ranker Training
Our BERT re-ranking experiment utilizes the pretrained "bert-base-uncased" model, hosted by HuggingFace. We use equal optimizer settings than for CoRT except for the learning rate, which we empirically set to  $5 \times 10^{-5}$ . We use a batch-size of 8 and accumulate the gradients of 16 batches.
B Appendix: Retrieval Examples
Table 6 shows top-1 retrieval examples of CoRT and BM25. The first query exemplifies the advantage of local interactions in the query encoder. We hypothesize, the query could be interpreted as a question about the density of aluminum although the term density was not included. The second query is an example, where BM25 works well due to favorable keywords in the passage. Although CoRT's top result is not labeled, it clearly is relevant to the query. Since the passage misses the keyword "insane", it is difficult to retrieve for a term-based model. We hypothesize, due to the terms "hallucinations" and "paranoia", CoRT is able to match the contexts in this example.
Table 6: Retrieval Examples with highlighted keywords. Ranks beyond the top-1000 are denoted with "n/a".
Query	Sample Passage	Label	Rank
BM25	CoRT
how much does aluminum weigh	Question and answer. how much does a western 14 ft. aluminum boat weigh? what is the weight difference between a 12 ft. and 14 ft. aluminum boat. 12 ft aluminum boat with no gear or anything would probably weigh about 115-150 lbs, 14 ft aluminum boat with no gear or anything would probably weigh about 250-300 lbs.	n/a	1	n/a
Quick Answer. One cubic inch of aluminum weighs 1.56 ounces. The metal sinks in water, but it is still relatively lightweight. The density of aluminum is 2.7 grams per milliliter. Aluminum is used as a metal foil, a conductor of electricity and in the construction of airplane fuselages.	True	735	1
how many days of no sleep until insane	Although there are many articles stating one can be declared legally insane if... How long must you go without sleep to be declared legally insane Each state would have different laws regarding the requirements of declaring... Is it true that after three days of complete sleep deprivation you are considered legally insane? Not after 3 days but any longer will.	True	1	n/a
The longest recorded time a human has ever gone without sleep is 18 days, 21 hours, and 40 minutes, which resulted in hallucinations, paranoia, etc. However most people can only last 4-6 days without stimulants, and about 7-10 days before the body will be unable to function and long term damage can be caused.	n/a	n/a	1
<sup>6</sup>https://huggingface.co/transformers/pretrained_models.html