Parsing Fixed Width Text Files with Pandas | by Amy Rask | Towards Data Science

A fixed width file is similar to a csv file, but rather than using a delimiter, each field has a set number of characters. This creates files with all the data tidily lined up with an appearance similar to a spreadsheet when opened in a text editor. This is convenient if you’re looking at raw data files in a text editor, but less ideal when you need to programmatically work with the data.
Fixed width files have a few common quirks to keep in mind:
When values don’t consume the total character count for a field, a padding character is used to bring the character count up to the total for that field.
Any character can be used as a padding character as long as it is consistent throughout the file. White space is a common padding character.
Values can be left or right aligned in a field and alignment must be consistent for all fields in the file.
A thorough description of a fixed width file is available here.
Note: All fields in a fixed width file do not need to have the same character count. For example: in a file with three fields, the first field could be 6 characters, the second 20, and the last 9.
Upon initial examination, a fixed width file can look like a tab separated file when white space is used as the padding character. If you’re trying to read a fixed width file as a csv or tsv and getting mangled results, try opening it in a text editor. If the data all line up tidily, it’s probably a fixed width file. Many text editors also give character counts for cursor placement, which makes it easier to spot a pattern in the character counts.
If your file is too large to easily open in a text editor, there are various ways to sample portions of it into a separate, smaller file on the command line. An easy method on a Unix/Linux system is the head command. The example below uses head with -n 50 to read the first 50 lines of large_file.txt and then copy them into a new file called first_50_rows.txt.
head -n 50 large_file.txt > first_50_rows.txt
The UniProt Knowledgebase (UniProtKB) is a freely accessible and comprehensive database for protein sequence and annotation data available under a CC-BY (4.0) license. The Swiss-Prot branch of the UniProtKB has manually annotated and reviewed information about proteins for various organisms. Complete datasets from UniProt data can be downloaded from ftp.uniprot.org. The data for human proteins are contained in a set of fixed width text files: humchr01.txt - humchr22.txt, humchrx.txt, and humchry.txt.
We don’t need all 24 files for this example, so here’s the link to the first file in the set:
https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/docs/humchr01.txt
A quick glance at the file in a text editor shows a substantial header that we don’t need leading into 6 fields of data.
Fixed width files don’t seem to be as common as many other data file formats and they can look like tab separated files at first glance. Visual inspection of a text file in a good text editor before trying to read a file with Pandas can substantially reduce frustration and help highlight formatting patterns.
Note: All code for this example was written for Python3.6 and Pandas1.2.0.
The documentation for pandas.read_fwf() lists 5 parameters:
filepath_or_buffer, colspecs, widths, infer_nrows, and **kwds
Two of the pandas.read_fwf() parameters, colspecs and infer_nrows, have default values that work to infer the columns based on a sampling of initial rows.
Let’s utilize the default settings for pandas.read_fwf() to get our tidy DataFame. We’ll leave the colspecs parameter to its default value of ‘infer’, which in turn utilizes the default value (100) of the infer_nrows parameter. These two defaults attempt to find a pattern in the first 100 rows of data (after any skipped rows) and use that pattern to split the data into columns.
There are several rows of file header that precede the tabular info in our example file. We need to skip them when we read the file.
None of the parameters seem ideal for skipping rows when reading the file. So how do we do it? We utilize the **kwds parameter.
Conveniently, pandas.read_fwf() uses the same TextFileReader context manager as pandas.read_table(). This combined with the **kwds parameter allows us to use parameters for pandas.read_table() with pandas.read_fwf(). So we can use the skiprows parameter to skip the first 35 rows in the example file. Similarly, we can use the skipfooter parameter to skip the last 5 rows of the example file that contain a footer that isn’t part of the tabular data.
pandas.read_fwf('humchr01.txt', skiprows=35, skipfooter=5)
The above attempt leaves the DataFrame a bit of a mess 😔:
Note: Since we’re using the default values for colspecs and infer_nrows we don’t have to declare them.
Part of the issue here is that the default colspecs parameter is trying to infer the column widths based on the first 100 rows, but the row right before the tabular data (row 36 in the file and shown in the column names above) doesn’t actually follow the character count patterns in the data table, so the inferred column widths are getting mangled.
If we’d set skiprows to 36 instead of 35, we’d have ended up with the first row of data pushed into the column names, which also mangles the inferred column widths. There’s no winning here without some additional cleanup. Let’s settle the column names issue with the names parameter and see if that helps.
Note: Using the names parameter means we are not allocating a row in the file to column names, so we as users have to make sure to account for the fact that skiprows must start at the first data row. So skiprows is set to 36 in the next example but it was 35 in previous examples when we didn’t use the names parameter.
pandas.read_fwf('humchr01.txt', skiprows=36, skipfooter=5, names=['gene_name', 'chromosomal_position', 'uniprot', 'entry_name', 'mtm_code', 'description'])
That’s better, but still a bit of a mess. Pandas inferred the column splits correctly, but pushed the first two fields to the index. Let’s fix the index issue by setting index_col=False.
pandas.read_fwf('humchr01.txt', skiprows=36, skipfooter=5, index_col=False, names=['gene_name', 'chromosomal_position', 'uniprot', 'entry_name', 'mtm_code', 'description'])
That looks good! The columns are split correctly, the column names make sense and the first row of data in the DataFrame matches the first row in the example file.
We relied on the default settings for two of the pandas.read_fwf() specific parameters to get our tidy DataFame. The colspecs parameter was left to its default value of ‘infer’ which in turn utilizes the default value of the infer_nrows parameter and finds a pattern in the first 100 rows of data (after the skipped rows) and uses that to split the data into columns. The default parameters worked well for this example file, but we could also specify the colspecs parameter instead of letting pandas infer the columns.
Just like with the example above, we need to start with some basic cleanup. We’ll drop the header and footer in the file and set the column names just like before.
The next step is to build a list of tuples with the intervals of each field. The list below fits the example file.
colspecs = [(0, 14), (14, 30), (30, 41), (41, 53), (53, 60), (60, -1)]
Note the last tuple: (60, -1). We can use -1 to indicate the last index value. Alternately, we could use None instead of -1 to indicate the last index value.
Note: When using colspecs the tuples don’t have to be exclusionary! The final columns can be set to tuples that overlap if that is desired. For example, if you want the first field duplicated: colspecs = [(0, 14), (0, 14), ...
pandas.read_fwf('humchr01.txt', skiprows=36, skipfooter=5, colspecs=colspecs, names=['gene_name', 'chromosomal_position', 'uniprot', 'entry_name', 'mtm_code', 'description'])
Once more we’ve attained a tidy DataFrame. This time we explicitly declared our field start and stop positions using the colspecs parameter rather than letting pandas infer the fields.
Reading fixed width text files with Pandas is easy and accessible. The default parameters for pandas.read_fwf() work in most cases and the customization options are well documented. The Pandas library has many functions to read a variety of file types and the pandas.read_fwf() is one more useful Pandas tool to keep in mind.