Package: indxr
⚡️ Introductionindxris a Python utility for indexing long files that allows you to quickly read specific lines dynamically, avoiding hogging your RAM.For example, given a 10M lines JOSNl file and a MacBook Pro from 2018, reading any specific line takes less than 10 µs, reading 1k non-contiguous lines takes less than 10 ms, reading 1k contiguous lines takes less than 2 ms, iterating over the entire file by reading batches of 32 lines takes less than 20 s (64 µs per batch). In other words,indxrallows you to use your disk as a RAM extension without noticeable slowdowns, especially with SSDs and NVMEs.indxrcan be particularly useful for dynamically loading data from large datasets with a low memory footprint and without slowing downstream tasks, such as data processing and Neural Networks training.For an overview, follow theUsagesection.🔌 Installationpipinstallindxr💡 Usagetxtjsonlcsv / tsvcallbackwrite / readPyTorch Dataset exampleTXTfromindxrimportIndxrindex=Indxr("sample.txt")# First line of sample.txtindex[0]# List containing the second and third lines of sample.txtindex[1:3]# First line of sample.txtindex.get("0")# List containing the third and second lines of sample.txtindex.mget(["2","1"])JSONlfromindxrimportIndxrindex=Indxr("sample.jsonl",key_id="id")# key_id="id" is by default# JSON object at line 43 as Python Dictionary# Reads only the 43th lineindex[42]# JSON objects at line 43, 44, and 45 as Python Dictionaries# Reads only the 43th, 44th, and 45th linesindex[42:46]# JSON object with id="id_123" as Python Dictionary,# Reads only the line where the JSON object is locatedindex.get("id_123")# Same as `get` but for multiple JSON objectsindex.mget(["id_123","id_321"])CSV / TSV / ...fromindxrimportIndxrindex=Indxr("sample.csv",delimiter=",",# Default value. Automatically switched to `\t` for `.tsv` files.fieldnames=None,# Default value. List of fieldnames. Overrides header, if any.has_header=True,# Default value. If `True`, treats first line as header.return_dict=True,# Default value. If `True`, returns Python Dictionary, string otherwise.key_id="id",# Default value. Same as for JSONl. Ignored if return_dict is `False`.)# Line 43 as Python Dictionaryindex[42]# Lines 43, 44, and 45 as Python Dictionariesindex[42:46]# Line with id="id_123" as Python Dictionaryindex.get("id_123")# Same as `get` but for multiple linesindex.mget(["id_123","id_321"])CustomfromindxrimportIndxr# The file must have multiple linesindex=Indxr("sample.something")# First line of sample.something in bytesindex[0]# List containing the second and third lines of sample.something in bytesindex[1:3]# First line of sample.something in bytesindex.get("0")# List containing the third and second lines of sample.something in bytesindex.mget(["2","1"])Callback (works with every file-type)fromindxrimportIndxrindex=Indxr("sample.txt",callback=lambdax:x.split())index.get("0")>>># First line of sample.txt split into a listWrite / Read IndexfromindxrimportIndxrindex=Indxr("sample.txt",callback=lambdax:x.split())index.write(path)# Write index to disk# Read index from disk, callback must be re-definedindex=Indxr.read(path,callback=lambdax:x.split())Usage example with PyTorch DatasetIn this example, we want to build a PyTorch Dataset that returns a query and two documents, one positive and one negative, for training a Neural retriever. The data is stored in two files,queries.jsonlanddocuments.jsonl. The first file contains queries and the second file contains documents. Each query has a list of associated positive and negative documents. UsingIndxrwe can avoid loading the entire dataset into memory and we can load data dynamically, without slowing down the training process.importrandomfromindxrimportIndxrfromtorch.utils.dataimportDataLoader,DatasetclassCustomDataset(Dataset):def__init__(self):self.queries=Indxr("queries.jsonl")self.documents=Indxr("documents.jsonl")def__getitem__(self,index:int):# Get query ------------------------------------------------------------query=self.queries[index]# Sampling -------------------------------------------------------------pos_doc_id=random.choice(query["pos_doc_ids"])neg_doc_id=random.choice(query["neg_doc_ids"])# Get docs -------------------------------------------------------------pos_doc=self.documents.get(pos_doc_id)neg_doc=self.documents.get(neg_doc_id)# The outputs must be batched and transformed to# meaningful tensors using a DataLoader and# a custom collator functionreturnquery["text"],pos_doc["text"],neg_doc["text"]def__len__(self):returnlen(self.queries)defcollator_fn(batch):# Extract data -------------------------------------------------------------queries=[x[0]forxinbatch]pos_docs=[x[1]forxinbatch]neg_docs=[x[2]forxinbatch]# Texts tokenization -------------------------------------------------------queries=tokenizer(queries)# Returns PyTorch Tensorpos_docs=tokenizer(pos_docs)# Returns PyTorch Tensorneg_docs=tokenizer(neg_docs)# Returns PyTorch Tensorreturnqueries,pos_docs,neg_docsdataloader=DataLoader(dataset=CustomDataset(),collate_fn=collate_fn,batch_size=32,shuffle=True,num_workers=4,)Each line ofqueries.jsonlis as follows:{"q_id":"q321","text":"lorem ipsum","pos_doc_ids":["d2789822","d2558037","d2594098"],"neg_doc_ids":["d3931445","d4652233","d191393","d3692918","d3051731"]}Each line ofdocuments.jsonlis as follows:{"doc_id":"d123","text":"Lorem ipsum dolor sit amet, consectetuer adipiscing elit."}🎁 Feature RequestsWould you like to see other features implemented? Please, open afeature request.🤘 Want to contribute?Would you like to contribute? Please, drop me ane-mail.📄 Licenseindxris an open-sourced software licensed under theMIT license.