Why are Data Scientists Afraid to Use Test Driven Development?
Source: tai_blog

Programming differs from Software Engineering and especially Data Science  but the question is what connects them and what should you strive to be?   Data Science teaches us how to deal with data in a proper way but that is not enough when building bigger systems such as data pipelines or ML ops. Learning to test your software is the first step towards becoming a software engineer.   In todays article  I would like to present the best practices for testing your software as well as great books that will advance your skills for the duration of your whole career.   This article is not just for Data Scientists but anyone who wants to upgrade their software engineering skills.   Lets jump right into it!   What is TDD?Test Driven Development is a methodology used when it comes to writing and testing code. It is a mindset in which you are writing the tests first (defining requirements) and then writing the code to fulfill those.   We cover all types of tests in this article but mostly focus on unit testing because that should be a standard. Unit testing describes tests that are run at the unit level and isolate components to be tested. They are straightforward  fast  and concise.   Tests are there to ensure the intended behavior is working properly. We define rules for it because it helps the workflow of the software engineer as well as the people reading the same code. Always think that code is written once and read at least ten times.   The beauty of code is writing it so simply and elegantly that it is a joy to read after with ease. But that is the hard part. One quote by Mario Fusco supports that:   The code you write makes you a programmer. The code you delete makes you a good one. The code you dont have to write makes you a great one. - Mario Fusco  Principal Software Engineer at Red Hat   What does this have to do with Data Science?Data Science comes into play here because in this realm of programming  we are not taught much about Software Engineering but statistics and parts of the data science life cycle such as data cleaning  processing  and visualization.   When creating data pipelines  writing clean code is really important. Ensuring the data flows are long-lasting  sustainable  and irresistible to different outside influences that can affect your software. Unexpected data types should not break your pipeline for starters. Rules are not easy to implement in your daily workflow  but in the long term  they will save your debugging time and production breakage at the most unexpected times.   To follow these rules  here is a point by point each rule that is important to follow within a Data Science environment. In this article  I will use Python and pytest to show the example with the simplest setup so you can follow along!   Define the problemPeople naturally start finding the solution to the problem they do not understand fully  which is usually the first mistake. Requirements should be a beginning step in any scenario so that you can even start thinking about a solution.   Understand what the client needs  put it clearly  and confirm with them.   Let me show you how to do that in a TDD environment with a small data science example converting milliseconds to a string with unit information:   1. Always start with a failing test for a function that we need to implement   Failing the test is important because once the function satisfies it  you will know that you achieved the requirement stated in the test itself.   Never write production code that does not satisfy the test you have written it for.   In this case  we have a class of tests called a test suite that holds all tests related to that function.The comment holds the information for the sphinx doc. The name of the test suite is a concept that satisfies the function we should implement. This helps later on when building documentation for other developers to read and find through your softwareThe description of the requirement should be simple and clear  telling the developers what the function is supposed to doAs a final touch  adding a simple example will give a great idea to a developer of what the function does more clearlyAn elegant solution to write multiple tests with the same assert command is to use parametrization by pytest and define inputs and outputs. This is called a happy path test with the most simple and straightforward test. For this test  we want units up to weeks worth of milliseconds.class Test_ms_to_str_with_unit:  .. concept:: Test_ms_to_str_with_unit :satisfies: S_ADD_ALL Large numbers in milliseconds are hard to read. They need to be converted to an easy-to-read string with unit. E.g. for 5.7 seconds: 5710 -> 5.7s  @pytest.mark.parametrize(value_in_ms str_output  [(None  '-')  (0  '0ms')  (5  '5ms')  (-5  '-5ms')  (50  '50ms')  (580  '0.6s')  (1000 * 5.71  '5.7s')  (1000 * 58.7  '59s')  (1000 * 59.3  '59s')  (1000 * 60 * 5.71  '5.7min')  (1000 * 60 * 58.7  '59min')  (1000 * 60 * 60 * 5.71  '5.7h')  (1000 * 60 * 60 * 18.7  '19h')  (-1000 * 60 * 60 * 18.7  '-19h')  (1000 * 60 * 60 * 24 * 5.71  '5.7d')  (1000 * 60 * 60 * 24 * 7 * 5.71  '5.7w')  (1000 * 60 * 60 * 24 * 7 * 18.7  '19w')]) def test_happy_path(self  value_in_ms  str_output): assert int_to_str_with_unit(value_in_ms) == str_output2. Implement the function with existing clear requirements.   Enter the function int_to_str_with_unit()  which uses set of rules defined under ms_rules as a list of tuples for minimum and maximum values of each unit.We go to infinity until the weeks limit has been breached.Go through the rules and find the fitting one  after that we compute the value by adding the unit information and building a string.ms_rules = [(1  100  '.0f'  'ms')  (1000  10  '.1f'  's')  (1000  60  '.0f'  's')  (1000 * 60  10  '.1f'  'min')  (1000 * 60  60  '.0f'  'min')  (1000 * 60 * 60  10  '.1f'  'h')  (1000 * 60 * 60  60  '.0f'  'h')  (1000 * 60 * 60 * 24  7  '.1f'  'd')  (1000 * 60 * 60 * 24 * 7  10  '.1f'  'w')  (1000 * 60 * 60 * 24 * 7  float('inf')  '.0f'  'w')] # this rule always appliesdef int_to_str_with_unit(value_in_int: float U+007C None  rules: list]) -> str:  converts an int with a unit to a human-readable string based on a list of rules  if value_in_int is None: return - for rule in rules: if (value_in_unit := value_in_int / rule[0]) < rule[1]: return f{value_in_unit:{rule[2]}}{rule[3]} return infAlthough this code is correct  keep in mind that some parts are unclear to read or cases unfulfilled and that we can improve it further. Lets do that next.   As simple as possible but not simplerThis idea sounds simple but hard to execute  keeping your code simple to read and hide complexity while executing it correctly is a masterpiece. That is why you start iterating. Write the first version of code that works  then from there  improve on the readability and reduce complexity if necessary.   Think of it as putting out ideas at first and brainstorming and then cleaning up and improving your idea that works. Dont be afraid to take that piece of code and remove it completely If you have a better idea. It is not the only goal that works. It has to work and be clean. In the end  just never go too simple so that solution does not work anymore.   Lets get back to the previous example. We made the rules for converting milliseconds to strings with units. Is that function really complete? What happens If the number is negative? How hard is that code to read? Lets fix that:   Introduce a data class called IntToStrRule defines the variables we can use in the method for enhanced readabilityAdding a simple check for negative numbers and adding it to the string at the end handles the negative numbers@dataclass class IntToStrRule: value_to_unit_divisor: int is_smaller_than: int U+007C float str_format: str unit_suffix: str ms_rules = [IntToStrRule(1  100  '.0f'  'ms')  IntToStrRule(1000  10  '.1f'  's')  IntToStrRule(1000  60  '.0f'  's')  IntToStrRule(1000 * 60  10  '.1f'  'min')  IntToStrRule(1000 * 60  60  '.0f'  'min')  IntToStrRule(1000 * 60 * 60  10  '.1f'  'h')  IntToStrRule(1000 * 60 * 60  60  '.0f'  'h')  IntToStrRule(1000 * 60 * 60 * 24  7  '.1f'  'd')  IntToStrRule(1000 * 60 * 60 * 24 * 7  10  '.1f'  'w')  IntToStrRule(1000 * 60 * 60 * 24 * 7  float('inf')  '.0f'  'w')] # this rule always appliesdef int_to_str_with_unit(value_in_int: float U+007C None  rules: list[IntToStrRule]) -> str:  converts an int with a unit to a human-readable string based on a list of rules  if value_in_int is None: return - if value_in_int < 0: value_in_int = abs(value_in_int) sign = - else: sign =  for rule in rules: if (value_in_unit := value_in_int / rule.value_to_unit_divisor) < rule.is_smaller_than: return f{sign}{value_in_unit:{rule.str_format}}{rule.unit_suffix} return infThis is much better. The code is readable  simple to read  and fulfills the requirements. Good job!   This gives a good baseline moving forward for further rules. Back to testing we go.   Name tests like those are your childrenReadability is not only in the body of the code but begins with the name of the functions. A good name can even make developers skip reading the simple functions. Write collapsable code.   There are only two hard things in Computer Science: cache invalidation and naming things.  Phil Karlton   This is obviously not just test function names but functions in general. For test function names  we describe the behavior we want to test with the function.   Lets say the implemented function processes data from the source and spreads it into an object called a component. Focus on the format and order of tests rather than the name itself because it is just an example. All tests are explained in the comments.   class Test_add_component:  Begin with test suite name same as the function name with Test_ in front so Test_add_component.  def test_happy_path(self  base_key):  This is the actual name of the test and shows intended use of the implemented function in the correct and simple way. Edge cases come after.  def test_an_empty_df_returns_the_same_df(self):  In Data Science we deal with empty data frames a lot so writing a test for such a edge case is useful.  def test_a_df_with_unit_information_adds_component_info(self):  This is first edge case test which shows specific intent. Format is usually *when this then that*.  def test_more_than_one_column_ending_with_unit_component_tree_raises_error(self):  When we want to show that our function has pruposeful limits  that can also be shown in tests. Again format is *when this then that*.  def test_speed_real_data(self):  This test is just an example that there are other tests than just unit tests. Using real data in testing is good to avoid unexpected cases of different data types for example. Benchmarking on real datasets can be beneficial to see if there are any significant changes in underlying libraries or data processed. To summarize  use the format of when this then that and use your test names to describe the behavior you want to achieve.   Cross the bridge of complexity when you get thereOnce you have fulfilled the requirements with your tests  writing a function should be simple  but sometimes we overengineer. Start simple and iterate.   Premature optimization is the root of all evil. - Donald Knuth   Sometimes  you dont need a function to run faster but code to be simpler. Imagine you save five milliseconds every day If you optimize a function  but it takes you three days to figure out a better algorithm.   Now  of course  you dont know that when you begin writing code  but that comes with experience. What you can do  on the other hand  is to find bottlenecks and think if the other option is worth implementing. Rough guessing also helps.   Chris Zimerman  in the book The Rules of Programming explains this in three lessons on optimization:   Dont optimize Make code simple and correct  dont worry about making it fast. If you need it to run fast  you will make it fast.Focus on bottlenecks Find parts of your code that are slow  so evaluating processor time is crucial. Check for potential corner cases and underlying bugs that you did not find in the first run. Measure the amount of data that is processed and reconfigure if needed. Optimize the newly found parts and start again.Dont worry too much We try to optimize too much and usually find big optimization mistakes early on  but the slight  elegant ones are harder and take time. Those come with experience. Dont worry too much  and try to learn from others.ConclusionThank you so much for reading this piece  and I hope it helped you understand the basic principles of test-driven development.   This topic is rather huge  and this article is not enough to handle everything at once  so I would love to recommend 6 lessons from Uncle Bob and especially this fourth video on TDD.   He talks about the rules of TDD as well as other programming basics that are still a standard of the industry.   That is all for today! If you have any questions  please send them my way!