Top 6 Python Tips for Data Scientists | by Kat Li | Towards Data Science

Coding is, and will remain to be an essential part of data science! It’s said that close to 70% of data scientists’ time is spent on coding, and I’m no exception. In this article, I will share 6 most useful and practical Python snippets that I have compiled from solving real analytics problems in my day-to-day work.
Now, let’s jump right in!
Dynamic execution with exec()
Dynamic execution with exec()
Scenario: You inherited a Python project from a colleague, and immediately noticed that those scripts all have a whopping 5000+ lines of code. The same chunks (of code) got copied and pasted multiple times! So, is there a more efficient option to go about code reusability?
Let’s explore the exec() function in Python. Simply put, it takes in a string or object code, and execute it as shown in this example,
a = 4 and b = 5a+b = 9
Even more handy? We can use exec(open().read()) to call and execute a file within the Python interpreter. For example,
With this powerful one-liner, data scientists can save programs that will be reused as standalone files, and execute them whenever needed within the main program. No code copying and pasting any more!
Being a cool functionality in Python, exec() has one pitfall to avoid — it does NOT return any value,
a = 4 and b = 5a+b = 9** Is the return from exec() is None? True **
As we can see, the output of the exec() function is None; hence, it cannot be used to store any values, which is equivalent to the sounce()function in R.
2. Operating system commands with {os} and {shutil}
Scenario: continue from our previous tip, now you want to check out the script before executing it. Don’t bother to double-click your mouse all the way through to open up the file? No problem, you can easily achieve this in Python directly without interrupting your train of thought.
Here, the os.startfile() function allows users to open up any type of files, including MS documents, Excels, R and SQL scripts.
Similarly, we can also delete a FILE using os.remove(“myFullFileName.ANYFORMAT”)
or delete the entire DIRECTORY using shutil.rmtree(“folderToBeRemoved”). where {shutil} is a Python module that offers a number of high-level file operations, particularly for file copying and removal.
Therefore, if you haven’t used {os} other than os.getcwd() or os.chdir() or if the {shutil} sounds unfamiliar, it’s time to check out their documents. You will definitely find useful commands or file system methods that make your coding easier. Here lists a few of my favorites,
os.listdir() or os.listdir(“someDirectory”) — list all files in the currently working directory or any user-specified directory, respectively;
os.path.join() — automatically create a path with elements in the arguments for later use, e.g., os.path.join(‘D’, ‘Medium’, ‘New Folder’) will return
 ‘D\\Medium\\New Folder’
os.makedirs() — create a directory;
shutil.copy2(“sourcePath”, “destinationPath”) or shutil.move(“sourcePath”, “destinationPath”) — copy or cut a file respectively.
3. One-liner: Nested list comprehension to get rid of the for loops
Scenario: this “simple” task we come across is to combine several lists into one big long list,
Surely, we can write five nested for loops to append each sublist to the final output list. But it’s smarter to turn to nested list comprehension for the most concise way,
4. Timer wrapped as a decorator
Scenario: while Python is recognized as one of the most effective programming languages, data scientists still need to check the runtime of our programs.
It’s not the hardest thing if we just implement a bare-bones Python timer for each function we want to monitor. However, if we code it as a decorator, we would make our timer much easier to be version-controlled and reused!
Here is how,
In this snippet,
the timer is wrapped in a timerWrapper function, which then is used as a decorator called prior to the main function;
The example main function is to return a request connecting to an URL, which is one of my previous blogs.
Running this code gives us the time elapsed,
Current function: getArtile Run Time: 1.6542516000008618Out[101]: <Response [200]>
Now, to time another function, all we need is to put the @timeWrapper in front of the function,
getMultiplication(3)101000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000Current function: getMultiplication Run Time: 0.00014700000065204222
5. Leverage the options system to customize your display
Scenario: as data scientists, we analyze data with {pandas} and {numpy} on a daily basis. When I first learned coding in Python, I was frustrated seeing this after reading my data into the IDE,
Clearly, data display is cut off both row-wise and column-wise, and the following code can fix it,
Here, we are explicitly setting the maximum columns, rows and column width to display/print in the console. There are numerous customizable options and settings in {pandas}, and similar operations are also available in {numpy} for arrays and matrix,
6. Reproduce your machine learning model results? Set seeds!
Scenario: Due to the stochastic nature of machine learning modeling process, we have all encountered the non-deterministic aspect of machine learning. This randomness results in our difficulty reproducing the same results across different runs. Consequently, it’s challenging to figure out whether an improvement in performance metrics is a result of successful model tuning or simply a different random training/testing sample.
Luckily the reproducibility can be achieved by setting the random seed throughout your model pipeline, provided that you do it correctly! How many times have you seen questions like “Getting a different result despite random seed defined” popping up on Stack Overflow? Well, how to appropriately set seeds should be in the first page of documentations, but it took me some time to dig it out.
I found that NOT every seed is defined the same in {numpy}, {sklearn}, {tensorflow} or {torch}. Thus, it’s a best practice to use a definitive function that sets all SEEDS for all your frameworks. For example,
Adding this tactical reset_random_seed() function to all necessary steps of your workflow, such as train-test split, model compile/training, and interpretation, will get you half way to full reproducibility. More detailed visibility into your experiments will finish the second half!
Hope you have learned some new functionalities or rediscovered some that you might have overlooked before. If you got inspired to use one or several of them in your own code, that would be fantastic! Leave comments below if you know of additional hidden gems of Python. 😃
Want more data science and programming tips? Use my link to sign up to Medium and gain full access to all my content.
Also subscribe to my newly created YouTube channel “Data Talk with Kat”,