Package: data4llm
Data4LLMThe simple and useful data process tool for LLMdata4llmdata4llmis a json & jsonline process tool, which runs well in millions number level, which facilitates the construction procession of millions of data to continue-pretrain and finetune your LLM. The current framework show below:installpip install data4llmAPISFTfromdata4llm.Data4LLMimportSFT1.For file level(1) merge filesmerge all the jsonlines files with shuffleimportglobfromdata4llmimportData4LLMfiles=glob("dir/*.jsonl")Data4LLM.merge_files(files=files)(2) split files to train and test filefromdata4llm.Data4LLMimportSFTSFT.split_train_test(file_input="data/test.jsonl",train_ratio=3/5)2. For sample levelEvery sample is a json with key-value like dict[str:str], for example:{"input":"hello!","output":"Hi, I'm an AI assistant, how can I help you?"}(1) shuffleshuffle all the samples in a file, it doesn't optimize the memory usage now, requiring to load all the data to memory in one timefromdata4llm.Data4LLMimportSFTSFT.shuffle(file_input="data/test.txt",file_output="result/sh_test.jsonl")def shuffle(cls, file_input, file_output):
    shuffle: shuffle all the data in input file. warning: it loads all the data in memory
    ile_input: input file path
    file_output: output file path(2) remove duplicated dataremove duplicate data by sim_hash. There are two functionremove_duplicate_BloomFilterandremove_duplicate.remove_duplicate_BloomFilter: remove duplicate data by sim_hash, which removes data by bloom filter, very fastfromdata4llm.Data4LLMimportSFTSFT.remove_duplicate_BloomFilter(file_input="data/test.jsonl",file_output="result/rm_dup_test.json",length=64)def remove_duplicate_BloomFilter(cls, file_input, file_output, max_row_limit=1000, skip_hash=False, length=64,
                                 log_path="result.log"):
    '''
        remove_duplicate : remove duplicate data by sim_hash, which removes data by bloom filter, very fast
        file_input: input file path with duplicated data
        file_output: result file path
        max_row_limit: the max data number in memory which is useful to save memory
        skip_hash: default false. it needed when call the function in first time, which is used to get the simhash in all the data
        length: the simhash length
        log_path: log file path
        :return: result data number , removed data number
    '''remove_duplicate: remove duplicate data by sim_hash, which compares data one by one, getting more accurate and finely result but costing massive timefromdata4llm.Data4LLMimportSFTSFT.remove_duplicate(file_input="data/test.jsonl",file_output="result/rm_dup_test.json",length=64)def remove_duplicate(cls, file_input, file_output, ratio=1, max_row_limit=1000, skip_hash=False, length=64,
                 log_path="result.log"):
    remove_duplicate : remove duplicate data by sim_hash, which compares data one by one, getting more accurate and finely result but costing massive time
    file_input: input file path with duplicated data
    file_output: result file path
    ratio: threshold for duplication, which is actually the distance of the two simhash value
    max_row_limit: the max data number in memory which is useful to save memory
    skip_hash: default false. it needed when call the function in first time, which is used to get the simhash in all the data
    length: the simhash length
    log_path: log file path
    :return: result data number , removed data number(3) applyThe most powerful function in this project, you can apply any process rule by it ,including:process property(rename,remove,add),process content(remove chars,replace chars),filtersample by some rules,derivedserval samples from a sample.
There are three typical ways:I.filter,II.process attributes,III.from one to serval:I. filtered by lengthFilter sample by returningNonedeffn(row:dict[str:str])->dict[str:str]:ifF.len(row)>1000:returnNonereturnrowSFT.apply(file_input="data/test.txt",file_output="result/result_test.jsonl",fn=fn)II. concat two properties into oneapply process to every samplefromdata4llm.Data4LLMimportSFT,Fdeffn(row:dict[str:str])->dict[str:str]:row['input']=row['instruction']+row['prompt']row.pop("instruction")row.pop("prompt")returnrowSFT.apply(file_input="data/test.txt",file_output="result/result_test.jsonl",fn=fn)III. from one to severalGenerate more samples from one sample by returning a list consisting of dictdeffn(row:dict[str:str])->List[dict[str:str]]:arrs=row['input'].split(";")[:-1]rows=[]temp_str=""fori,iteminenumerate(arrs):ifi%2!=0:output=item.split(":")[0]temp_row={"input":temp_str+"assistant:","output":output}rows.append(temp_row)else:temp_str+=item+";"returnrowsSFT.apply(file_input="data/test.txt",file_output="result/result_test.jsonl",fn=fn)The apply functiondef apply(cls, file_input, file_output, fn, max_row_limit=1000, json=None):
    apply_property: apply the json row one by one, including: rename property, remove property, apply content(remove chars, replace chars)
     file_input: input file path
     file_output: output file path
     fn: apply function
     max_row_limit: default=1000, every step to write file and max data num in memory
     json: default=None, it determines json or jsonline, or True/False(4) show_exampleIt is very useful to show the result before actually conduct it byshow_example:fromdata4llm.Data4LLMimportSFTdeffn(row):row['input']=row['input'].replace("https://www.baidu.com")returnrowSFT.show_example(file_input="data/test.txt",fn=fn)examples:##### No 1 #####
== Before ==
{'input': 'welcome to https://www.baidu.com #LLM world', 'output': 'I like #LLM'}
== After ==
{'prompt': 'welcome to  LLM world', 'chosen': 'I like LLM'}
##### No 2 #####
== Before ==
{'input': 'hello!', 'output': "Hi, I'm an AI assistant, how can I help you?"}
== After ==
{'prompt': 'hello!', 'chosen': "Hi, I'm an AI assistant, how can I help you?"}def show_example(cls, file_input, fn, json=None, s=0, e=5):
    '''
    file_input: 
    fn: 
    json: if the file is json or jsonline, default None means it decided by the postfix of th file_input 
    s: default 0 the start row num
    e: default 5 the end row num
    :return: None
    '''PTfromdata4llm.Data4LLMimportPT(1)  show_propertiesshow the json structuredefshow_properties(cls,files,s=0,e=5):'''show the json structure:param files::param s::param e::return:'''(2) parse_pagesparse the semi structure json and parse all the token needed together for PTdefparse_pages(cls,files,fn,output_dir):'''parse the semi structure json and parse all the token needed together fot PT:param files::param fn::param output_dir::return:'''(3) merge_filesmerge all the txt filesdefmerge_files(cls,files,output_file="merge_file.txt",max_limit_num=100):'''merge all the txt files:param files::param output_file::param max_limit_num::return:'''(4) split_train_testsplit a file into train and test filesdefsplit_train_test(cls,file_input,train_test_ratio,file_train_output="train.txt",file_test_output="test.txt"):'''split a file into train and test files:param file_input::param train_test_ratio::param file_train_output::param file_test_output::return:'''(5) samplesampleFA tool class with some useful functionsfromdata4llm.Data4LLMimportF(1) countget the sample number of a filedefcount(cls,file_input):"""get the sample number of a file:param file_input::return:"""(2) functions used inapplyfnrename(): rename the property of every samplerepalce(): replace the chars in a json or a property in the jsonlen(): get the length of the json (only values) of part of json (specify the property like "chosen" only {"chosen"})defrename(cls,row,mapping:dict[str:str])->Nonedefreplace(cls,row,pattern,repl,property=None)->Nonedeflen(cls,row,property=None)->int: