3 Python Tricks to Read, Create, and Run Multiple Models Automatically | by Khuyen Tran | Towards Data Science

When putting your code into production, you will most likely need to deal with organizing the files of your code. It can be really time-consuming to read, create, and run many files of data. This article will show you how to automatically
Loop through files in a directory
Create nested files if they do not exist
Run one file with different inputs using bash for loop
These tricks have saved me a lot of time while working on my data science projects. I hope you will find them useful as well!
If we have multiple data to read and process like this:
├── data│   ├── data1.csv│   ├── data2.csv│   └── data3.csv└── main.py
we can try to manually read one file at a time
import pandas as pd def process_data(df):   passdf = pd.read_csv(data1.csv)process_data(df)df2 = pd.read_csv(data2.csv)process_data(df2)df3 = pd.read_csv(data3.csv)process_data(df3)
This works but not efficiently when we have more than 3 data. If the only thing we change in the script above is the data, why not use the a for loop to access each data instead?
The script below allows us to loop through files in a specified directory
data/data3.csvdata/data2.csvdata/data1.csv
Here are the explanations for the script above
for filename in os.listdir(directory) : loop through files in a specific directory
if filename.endswith(".csv") : access the files that end with ‘.csv’
file_directory = os.path.join(directory, filename) : join the parent directory (‘data’) and the files within the directory.
Now we can access all your files within the ‘data’ directory!
Sometimes we might want to create nested files to organize our code or model, which makes it easier in the future for us to find them. For example, we might use ‘model 1’ to specify a specific feature engineering.
While using model 1, we might want to use different types of machine learning models to train our data (‘model1/XGBoost’).
While using each machine learning model, we might even want to save different versions of the model because of the difference in hyperparameters used for the model.
Thus, our model directory can look as complicated as below
model├── model1│   ├── NaiveBayes│   └── XGBoost│       ├── version_1│       └── version_2└── model2    ├── NaiveBayes    └── XGBoost        ├── version_1        └── version_2
It might take a lot of time for us to manually create a nested file for every model we create. Is there a way that we can automate this process? Yes, with os.makedirs(datapath).
Run the file above and you should see the nested file ‘model/model2/XGBoost/version_2’ created automatically!
Now you can save your model or data to the new directory!
What if we want to run one file with different parameters? For example, we might want to use the same script to predict data using different models.
If one script takes a long time to run and we have multiple models to run, it will be time-consuming for us to wait for the script to finish running then run the next one. Is there a way that we can tell the computer to run model 1, 2, 3,.., 10 with one command line then go and do something else.
Yes, we can with for bash for loop. First, we use sys.argv to enable us to parse command-line arguments. You can also use tools such as hydra if you want to overwrite your config file on the command line.
>>> python train.py XGBoost 1Loading model from model/model1/XGBoost/version_1 for training
Sweet! We have just told our script to use the model XGBoost, version 1 to predict the data on the command line. Now we can use the bash for loop to loop through different versions of the model.
If you can do a for loop with Python, you can also do that on the terminal like below
$ for version in 2 3 4> do> python train.py XGBoost $version> done
Type Enter to separate the lines
Output:
Loading model from model/model1/XGBoost/version_1 for trainingLoading model from model/model1/XGBoost/version_2 for trainingLoading model from model/model1/XGBoost/version_3 for trainingLoading model from model/model1/XGBoost/version_4 for training
Now you can do other things while letting your script run using different models! How convenient is that!
Congratulations! You have just learned how to automatically read and create multiple files at once. You also learned how to run one file with different parameters. The time you spend manually reading, writing, and running files can now be saved for more important tasks.
If there are some parts in the articles that you are confused about, I created concrete examples in this repo.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: