Package: linereader
Overviewlinereader is a python package that gives the user the ability to access
files with ease. The linereader package offers several new powerful ways
of using files.Two main new types of file handles are added to linereader:1-copen, a cache based solution to random file access and dynamic
processing2-dopen, a slower but universal way of random file access and
dynamic processingRandom file access and processing with a cachelinereader was meant as a direct substitute to python’s built-in
linecache module. With linereader, cached entries are loaded using less
memory, and are around 12% faster to access than those of linecache.
There are extra utility functions added to linereader to aid in the
manipulation of the global cache.If one wants to get a upgrade of linecache, linereader offers a
polymorphicgetlinefunction, where:from linecache import getlinecan be replaced by:from linereader import getlineand both behave the same way, loading a file’s contents into cached
memory.An example of this usage would be as follows:from linereader import getline

filename = 'C:/Python34/testfile.txt'

line_1 = getline(filename, 1)
line_2 = getline(filename, 2)

print(line_1, line_2)In addition togetline, linereader also containsgetonce, andcopen, which are used as solutions to cache based file access.Random file access and processing without loading into memoryThe problem with file accessing methods that load the entire file into a
cache, is that they only work on small files. Usually, a 5GB file cannot
be loaded into memory without the python interpreter crashing. Even if a
file can be loaded, it slows down the session, and eats up useful
memory. A new file handle that was added into linereader,linereader.dopen, works around this problem and can access any line
from any size text/logging/data file with consistency. The speed to
which the file can be accessed is proportional to the amount of
characters that are being read. There exists a slight python overhead
when accessing any file line, that takes around 31 microseconds. Using a
10 GB test file, a line consisting of one character was returned in 31
microseconds, and a line containing 135 characters was returned in 97
microseconds.dopen’s special internals allow for near-identical return speeds on same
length lines within the same file. This means that if file a was loaded
using dopen, and lines 368 and 290 both contained the same amount of
characters, they would take almost the same exact time to return. The
way that the dopen handle was made, allows for the ability to quickly
jump from one position in a file to the next. Conventional methods of
reading from a file have to iterate through all the characters or lines
and silently read the content that the user doesn’t want, to pass over
and get to the the content that they need.A simple example of dopen’s usage results as follows:import linereader

filename = 'C:/Python34/NEWS.txt'
file = linereader.dopen(filename)

header = file.getline(1) + file.getline(2)
line_500 = file.getline(500)
line_38 = file.getline(38)
from_38_to_500 = file.getlines(38, 500)The usage of dopen gets very advanced, and is actually completely
polymorphic with the regularopen()handle:import linereader

filename = 'C:/Python34/README.txt'
file = linereader.dopen(filename)

file.seek(50)
chars = file.read(10)
file.seek(1337)

chars += file.read(80)
chars += file.readline()

rest = file.readlines()In addition, dopen also offers powerful methods for the navigation of
the file pointers:from linereader import dopen
file = dopen('C:/Python34/README.txt')

file.seekline(58)
line_58 = file.readline()
next_10_lines = file.readnext(10)
line_67 = file.getline(67)ContactIf you have any questions or issues regarding linereader, please contact me at:-npandolfi@wpi.edu