power query functions, power query functions list,power query functions table, power bi functions, power query functions custom, functions apply | Towards Data Science

Cleaning, wrangling and summarizing. I don’t know about you but I enjoy this process of data analysis. Power Query is the main tool I use for this. I’ll be writing more about Power Query in the future.
I clean because the data is dirty. I clean because I want good data for analysis. I wrangle to get the right “shape” for joins to work. It’s dreadful if you have to clean data manually and it’s prone to cleaning errors. Imagine error on top of errors? What will you really be analyzing?
Don’t worry. Functions — much like those cute little robots on top can come to the rescue.
What exactly is a function anyways? A quick google search gives this definition.
“A function is a block of organized, reusable code that is used to perform a single, related action. ... Different programming languages name them differently, for example, functions, methods, sub-routines, procedures, etc.”
The keyword here is reusable. It has to be reusable for it to be a function. That means you can use this code and apply it to not just one file but other files as well. I think of a function as something that takes inputs, performs a custom set of procedures, and gives an output.
A lot of what you use today, for example, the sum function in Power BI to the joins are functions.
For example — take a look at this function in M (Power Query Language)
The problem is...sometimes we don’t have functions that are available to us that quite meet our needs. We want to apply a custom set of procedures to various files. This is why and when we have to use it.
Let’s go through how to create functions in Power BI in Power Query. With all the cleaning and wrangling in life, I’m sure this will be very useful to us.
As the stick figure mentioned, We have several different files across the years. we need to perform these same actions across all the files.
We don’t want to do this procedure to each one of the files but we want the function to do this for us.
To get started, let’s load in all our files in a folder.
After you’ve loaded all your files, you should see them displayed as below.
Let’s start by creating a “Duplicate” of the main file — we want to build a function from this duplicate.
Now that there is a duplicate, we keep only the first row to build the function.
Power BI keeps only the first row. Nice.
Here is a neat and important trick — you can type in [Content] which refers to the column and {0} which refers to the actual “cell” itself. If you have more of a technical understanding, I am referring to the column and returning a list, then referring to the list value 0 to return the binary file.
If you don’t like typing, you can also click on “drill down” on the binary file itself.
Power BI neatly loads the file without those automatic transformations.
Now let’s do our quick summary using group by. You can find “Group By” under the Transform Ribbon on the top of the pane.
Here is our result for one of the files.
This is exactly what we need.
Hmm... stick figures.
Not done yet, let’s make it into an actual function.
You can do this by going to the Advance Editor option under the Transform Ribbon.
Power BI records all the steps and turns it into an M script for us. If you are familiar with Excel, it’s like a recorded macro. we just need to change the reference so that it can be applied to other files.
You don’t need to know how to type out all the M code below. The beauty of Power BI is that it records all these steps for you. I will be writing a bit more about how to read M scripts in the near future.
This is the current script.
let    Source = Folder.Files("MY_DRIVE"),    #"Removed Top Rows" = Table.Skip(Source,1),    #"Removed Other Columns" = Table.SelectColumns(#"Removed Top Rows",{"Name", "Content"}),    #"Kept First Rows" = Table.FirstN(#"Removed Other Columns",1)[Content]{0},    #"Imported CSV" = Csv.Document(#"Kept First Rows",[Delimiter=",", Columns=5, Encoding=65001, QuoteStyle=QuoteStyle.None]),    #"Promoted Headers" = Table.PromoteHeaders(#"Imported CSV", [PromoteAllScalars=true]),    #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Year", Int64.Type}, {"Sex", type text}, {"Name", type text}, {"Job Title", type text}, {"Salary", Int64.Type}}),    #"Grouped Rows" = Table.Group(#"Changed Type", {"Year", "Sex"}, {{"Average Salary", each List.Average([Salary]), type nullable number}}),    #"Rounded Off" = Table.TransformColumns(#"Grouped Rows",{{"Average Salary", each Number.Round(_, 0), type number}})in    #"Rounded Off"
Don’t let this overwhelm you, the only thing we care about is the Csv.Document part.
Let’s remove everything above the Csv.Document part and declare our variable. I chose (X as binary) because we want this function to have a binary input.
Remember all those csv files on the folder? They are binary files.
This is our function!
Now let’s apply this to our files. Let’s “invoke” our custom function. You can find “invoke” under the “add column” ribbon.
Sounds like something out of Harry Potter right?
Here it is!
Let’s hit expand for our final table.
Ta-da!
Male and Female by average salary by year.
Now, this is a simple data set but you can also build in joins and other much more complex transformations into a function and apply it to all the files.
TL:DR? I built this for you.
Hope you’ve enjoyed this article.
If you’d like to watch a video instead — Curbal is a channel that has really taught me a lot about functions and of course Power BI.
Stay safe and hope this helps you with your journey with data!