Question: I'm sure there is a better way for this but I am drawing a blank. I have a CSV file in this format. The ID column is sorted so everything is grouped together at least: Text ID this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text2, BBBB this is sample text2, BBBB this is sample text2, BBBB this is sample text3, CCCC this is sample text4, DDDD this is sample text4, DDDD this is sample text5, EEEE this is sample text5, EEEE this is sample text6, FFFF this is sample text6, FFFF  What I want to do is split the CSV fast across X amount of smaller CSV files fast. So if X==3, then AAAA would go into "1.csv", BBBB would go into "2.csv", CCCC would go into "3.csv" and the next group would loop back around and go into "1.csv". The groups vary in size so a hardcoded split by numbers won't work here. Is there a faster way to split these reliably then my current method which just uses Pandas groupby in Python to write them?  file_ = 0 num_files = 3 for name, group in df.groupby(by=['ID'], sort=False): file_+=1 group['File Num'] = file_ group.to_csv(file_+'.csv',index=False, header=False, mode='a') if file_ == num_files: file_ = 0  This is a python based solution but I am open to stuff using awk or bash if it gets the job done. EDIT: For clarification, I want the groups split across a fixed amount of files I can set. In this case, 3. (So x = 3). The first group (AAAA) would go into 1.csv, the 2nd into 2.csv, the third into 3.csv and then for the fourth group, it would loop back and insert it into 1.csv. etc. Example output 1.csv: Text ID this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text4, DDDD this is sample text4, DDDD  Example output 2.csv: Text ID this is sample text2, BBBB this is sample text2, BBBB this is sample text2, BBBB this is sample text5, EEEE this is sample text5, EEEE  Example output 3.csv: Text ID this is sample text3, CCCC this is sample text6, FFFF this is sample text6, FFFF 

Answer: Using any awk in any shell on every Unix box: $ cat tst.awk NR==1 { hdr = $0 next } $NF != prev { out = (((blockCnt++) % X) + 1) ".csv" if ( blockCnt <= X ) { print hdr > out } prev = $NF } { print > out }   $ awk -v X=3 -f tst.awk input.csv   $ head [0-9]*.csv ==> 1.csv <== Text ID this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text, AAAA this is sample text4, DDDD this is sample text4, DDDD ==> 2.csv <== Text ID this is sample text2, BBBB this is sample text2, BBBB this is sample text2, BBBB this is sample text5, EEEE this is sample text5, EEEE ==> 3.csv <== Text ID this is sample text3, CCCC this is sample text6, FFFF this is sample text6, FFFF  If X was some large enough number that you exceed your system limit for concurrently open files and you start getting a "too many open files" error then you'd need to use GNU awk as it handles that internally or change the code to only have 1 file open at a time: NR==1 { hdr = $0 next } $NF != prev { close(out) out = (((blockCnt++) % X) + 1) ".csv" if ( blockCnt <= X ) { print hdr > out } prev = $NF } { print >> out }  or implement your own way of managing how many files are open concurrently.  EDIT: here's what the suggestion by @PaulHodges in the comments would result in a script like: NR == 1 { for ( i=1; i <= X; i++ ) { print > (i ".csv") } next } $NF != prev { out = (((NR-1) % X) + 1) ".csv" prev = $NF } { print > out }