Unix & Linux: cat line X to line Y on a huge file (6 Solutions!!)
Channel: Roel Van de Paar
Date: 2020-08-07

Unix & Linux: cat line X to line Y on a huge file


The Question: Say I have a huge text file (＞2GB) and I just want to cat the lines X to Y
(e.g. 57890000 to 57890010).
From what I understand I can do this by piping head into tail or viceversa,
i.e.
head -A /path/to/file | tail -B
or alternatively
tail -C /path/to/file | head -D
where A,B,C and D can be computed from the number of lines in the file, X and
Y.
But there are two problems with this approach:
 1. You have to compute A,B,C and D.
 2. The commands could pipe to each other many more lines than I am
 interested in reading (e.g. if I am reading just a few lines in the
 middle of a huge file)
Is there a way to have the shell just work with and output the lines I want?
(while providing only X and Y)?

Solutions: Please watch the whole video to see all solutions, in order of how many people found them helpful

== This solution helped 17 people ==
The most orthodox way (but not the fastest, as noted by https://
unix.stackexchange.com/a/47424/16705 above) would be to use sed.
In your case:
X=57890000
Y=57890010
sed -n -e "$X,$Y p" -e "$Y q" filename
The -n option implies that only the relevant lines are printed to stdout.
The p at the end of finishing line number means to print lines in given range.
The q in second part of the script saves some time by skipping the remainder of
the file.

== This solution helped 25 people ==
The head | tail approach is one of the best and most "idiomatic" ways to do
this:
X=57890000
Y=57890010
＜ infile.txt head -n "$Y" | tail -n +"$X"
As pointed out by Gilles in the comments, a faster way is
＜ infile.txt tail -n +"$X" | head -n "$((Y - X))"
The reason this is faster is the first X - 1 lines don't need to go through the
pipe compared to the head | tail approach.
Your question as phrased is a bit misleading and probably explains some of your
unfounded misgivings towards this approach.
 * You say you have to calculate A, B, C, D but as you can see, the line
 count of the file is not needed and at most 1 calculation is necessary,
 which the shell can do for you anyways.
 * You worry that piping will read more lines than necessary. In fact this
 is not true: tail | head is about as efficient as you can get in terms of
 file I/O. First, consider the minimum amount of work necessary: to find
 the X'th line in a file, the only general way to do it is to read every
 byte and stop when you count X newline symbols as there is no way to
 divine the file offset of the X'th line. Once you reach the *X*th line,
 you have to read all the lines in order to print them, stopping at the
 Y'th line. Thus no approach can get away with reading less than Y lines.
 Now, head -n $Y reads no more than Y lines (rounded to the nearest buffer
 unit, but buffers if used correctly improve performance, so no need to
 worry about that overhead). In addition, tail will not read any more than
 head, so thus we have shown that head | tail reads the fewest number of
 lines possible (again, plus some negligible buffering that we are
 ignoring). The only efficiency advantage of a single tool approach that
 does not use pipes is fewer processes (and thus less overhead).

== This solution helped 130 people ==
I suggest the sed solution, but for the sake of completeness,
awk 'NR ＞= 57890000 && NR ＜= 57890010' /path/to/file
To cut out after the last line:
With thanks & praise to God, and with thanks to the many people who have made this project possible! | Content (except music & images) licensed under cc by-sa 3.0 | Music: https://www.bensound.com/royalty-free-music | Images: https://stocksnap.io/license & others | With thanks to user Stephane Chazelas (https://unix.stackexchange.com/users/22565), user Pawel Rumian (https://unix.stackexchange.com/users/16705), user Kevin (https://unix.stackexchange.com/users/11750), user jw013 (https://unix.stackexchange.com/users/9537), user Gilles 'SO- stop being evil' (https://unix.stackexchange.com/users/885), user Doolan (https://unix.stackexchange.com/users/87207), user Amelio Vazquez-Reina (https://unix.stackexchange.com/users/4531), and the Stack Exchange Network (http://unix.stackexchange.com/questions/47407). Trademarks are property of their respective owners. Disclaimer: All information is provided "AS IS" without warranty of any kind. You are responsible for your own actions. Please contact me if anything is amiss at Roel D.OT VandePaar A.T gmail.com.