Using R to Merge the CSV Files in Code-Point Open Into One Massive File | by Jamie Whyte | Towards Data Science

If you have a dataset that you have postcodes for — eg customer data, or asset information, one of the most useful things you can do with it is put that data onto a map. To do that, each record needs to be geo-referenced. Ordnance Survey have an open dataset, called Code-Point Open, which you can get from their Open Data site. This dataset allows you to lookup postcodes in your own data to find their co-ordinates, so you can then put them on a map.
The dataset from Ordnance Survey is contained in a zip file, and when you extract it, it consists of 120 csv files. This can be a pain in the backside to deal with, so I’m going to show how to use R (actually R Studio) to stick all these csv files together into one. There are other ways of doing this — manually (😱), writing a batch file, MS Visual Basic, etc, but this is an elegant and quick way to get it done.
To start with, open up R Studio, and make a new project. Create a new folder in the project folder, and call it ‘temp’. You now need to copy the 120 csv files from Ordnance Survey into this folder.
After this, in the R Studio console, we need to set the working directory of this project to that temp folder:
setwd(‘temp/’)
Then, we need to pass all the filenames of everything in that folder into a variable:
filenames <- list.files(full.names=TRUE)
This will go through the files in that folder, pick out the filenames, and pass them into the variable ‘filenames’. We’ll now use those file names to iterate through the files, extracting the data from each csv file, and combining it into one list (Note that this will take a few seconds to complete):
All <- lapply(filenames,function(i){read.csv(i, header=FALSE, skip=4)})
This pushes the data into the variable 'All'. If we look in the ‘Environment’ tab, we can see that this is a ‘Large list’ containing 120 elements, and is 156Mb in size.
To use the datasets, either to do matching in R, or to save as its own csv file, we need to turn it into a dataframe. To do this, we can run the following:
df <- do.call(rbind.data.frame, All)
To check the contents of the dataframe, we can use the head function to show the first few rows of data:
head(df)
Which give us this:
This looks good (although it would probably be good practice to rename to columns to something more meaningful). V1 has the postcode, and V3 and V4 the British Co-ordinate System references. Finally, to save this big postcode list as a new csv file we can use write.csv:
write.csv(df,”all_postcodes.csv”, row.names=FALSE)
This will create a new (massive) csv file in the temp folder in the R Studio project folders. From here, you can either merge the postcode file with your own data to geo-reference it within R, or you can go and do vlookups in Excel.
Note that this exercise works because each of the csv files has the same shape, in terms of the number of columns of data. This kind of process gets more complicated for different datasets.
Here’s all the code in one go:
setwd(‘temp/’)filenames <- list.files(full.names=TRUE)All <- lapply(filenames,function(i){read.csv(i, header=FALSE, skip=4)})df <- do.call(rbind.data.frame, All)write.csv(df,”all_postcodes.csv”, row.names=FALSE)