Appending CSV files. Power BI, R, Power Shell or CMD | by Peter Hui | Towards Data Science

Have you ever had to combine multiple CSV files? I’m sure it’s a task every data analyst or data scientist must do at some point, and a strange task really. Why are they separated in the first place? Isn’t it better to connect to the source? Sometimes the best solution may not always be possible.
If you find yourself opening Excel, copying files and juggling 10 different workbooks, then this article is for you. I’ll show you different ways of appending files. Not only will you save time, but it will save you from errors as well. Trust me, I used to do this before I discovered Power BI and R.
If you don’t have to do this and you are well acquainted with Power BI and R, then I’m sure you will find some interesting scripts with command line and Power Shell.
Let’s get started.
I have 6 files from the image here and they have 29 columns in each file.
These are all located in one folder. The end result I am trying to get is a bigger file that combines all 6 into one. The number here doesn’t matter, it can be 6 or 100. The process is still the same :)
Why are we combining these? Reasons varies. Sometimes it’s easier to work with a big table for a simple analysis. Sometimes it maybe because of a data type issue and files cannot be loaded. In whatever case, there are different tools to append these files.
With Power BI can simply go to Power BI and use the connect to the folder to connect it all.
Here you go — all the files in Power Query for you to combine and load.
Sometimes Power BI may find a data error and won’t be able to load these files. Here is an article on some of the errors and how to fix them.
Power BI won’t be able to load all these files because of a column error. Sometimes one single cell may have letters but instead Power BI has categorized that column as numeric. This will cause and error.
As a quick tip, delete the “change types” step when loading files. It seems to cause errors more than it is useful. If I remove this step, you can see that Power BI is able to load it all.
PRO: No code, just clicks. Simple navigation and GUI. Allows you to do other type of analysis by table and functions using Table functions and custom functions.
CON: Slow, may not work and require trouble shooting if there are data type errors. Frustrating if you have a large amount of files and discovering a type error after waiting.
Here I can use the package call VROOM, get all the file names in the folder and combine it all.
In R Studio type in
library(vroom)setwd("Your folder directory")Files = dir_ls(glob = "*.csv")Files_Combined = vroom(Files)
The script here says to set your working directory, using dir_ls, filter for anything that ends with “.csv” and save it into a variable call files. Then vroom() reads the “Files” and assigns it to the end variable call Files_Combined.
Here — it didn’t exactly work for me because one of the file has 150 columns.
To correct this, you can use map instead. If you are not familiar with R, map applies a function over the variable provided. In this case it is the Files — which contains the names of the CSV files we want to load. The function here would be vroom() which is the load function.
library(vroom)setwd("Your folder directory")Files = dir_ls(glob = "*csv")Files_Combined = map(Files, ~vroom(.x))
Here all the files will be loaded but in list format. You can then choose the files you want by sub setting. You can use Files_Combined[1:3] or refer to the actual file itself using names Files_Combined[“your_file_name”].
PRO:Very fast! Also really like the sound of it :) Able to load all the files in a list format in 5 seconds.
CON: Assumes you have same number of column headers. I have to do some tweaking to see which file has extra columns before running vroom. Cannot bind rows if one column in data frame one is numeric and the other one is character.
Here is the script to combine these files into a CSV document. You can type in “PowerShell ISE” in the start up menu and PowerShell will be loaded.
A blue screen will appear — don’t worry it’s not the blue screen of death. :)
Here you can type in.
cd "Your folder directory"Get-ChildItem -Filter *csv | Get-Content | Add-Content "Combined.csv"
Get-ChildItem will list all the files that are in that folder. If you wish, you can even use -Recurse and it will list all the files in the relevant sub-folders as well. Get-Content takes the content of each of the CSVs and Add-Content will add the content into a new file call “combined.csv”.
Nifty :)
PRO: Reliable. Doesn’t throw type errors. It just does what you tell it to do.
CON: It’s great, no issues. Happy to see that it didn’t throw any errors! Can be both a good and bad thing, the extra column didn’t get appended into the new file.
This one is my favorite. Simply type in CMD in your start up menu and a black screen will appear. Don’t worry it’s just the command prompt.
cd "your folder directory"copy *csv combined.csv
This simple line will copy all the CSV files in the folder you have created into a new file call “combined.csv”.
Here it says to copy everything that ends with “csv” and put them into a document call “combined.csv”.
The great thing about this is you can add in grep to filter for values you want before combining.
grep "your_item" *csv >> item.csv
Here you “grep” your item from all the CSV files in that folder and export it into a new file call “item.csv”
I do like this one :)
Image you have to search for one single item, let’s say an account number in 200 files? grep with CMD is your friend.
PRO: Very fast and no issues. Copied everything in 3 seconds.
CON: A bit cryptic if you don’t know what you are reading.
There are different tools available to append CSVs.
The major headache, at least for me, is dealing with type issues. Both PowerShell and CMD can by pass all that and provide you with a table for analysis.
If you are just looking to append files, do some filtering and work on a combined document, use PowerShell or CMD. You can use Power BI or R to do the analysis after without worrying about the data format issues. I haven’t tried CSVKIT yet, but when I do, I’ll sure let you know.
Someone once asked me, “Why do you bother writing? You make <5 dollars an article anyways”. It’s a good question. The truth is, I learned a lot from others. I think it’s important to share knowledge. No one got to where they are today without another’s help. We all rely on each other, both directly and indirectly.
Thanks! Stay safe and good luck to your data journey :)