Separation of Concerns in Real Life Apps
By: Vinicius Ronconi
Tags: Coding, Software Engineering, Software Development

If you’re developing a web site or an app, there is a chance that you are using a framework. Python + Django/Flask/Tornado, Node + Express, Ruby on Rails, PHP Laravel, Java Spring or any other option available in the market.

All of them are great! If you follow the most known tutorials available on the internet, there is a good chance that you create a code that helps new developers understand what is going on in the system since you’re following a well-known architecture in the community.

But is this enough to ensure we’re creating an easily maintainable software? Maybe it is not always true. These tutorials were created to ensure you’re using the framework in the best way possible. From the framework creators point of view, it makes sense. It ensures the framework works at its best. But what if the framework does not continue to evolve on the way to match your system needs?

I’ve been working in the software development industry long enough to work on projects to convert mainframe to Borland Delphi. And from Delphi to Microsoft .NET. On the front-end, it can be even more intense: from plain JavaScript to some ancient framework. From primitive frameworks to Backbone. From Backbone to Angular and from Angular to React.

These migrations are never easy. But if there is any advice to make it a bit less stressful, it comes from Uncle Bob on his “Clean Architecture” book: do not marry your framework!

It means that we should not tie all our classes to our current framework. We’re committing to a framework that does not have any commitment with our project. What happens if the framework does not develop in the way we would like to see? Or if it does not develop at all? From the framework side, there is no big deal. But it can be a tragedy for our project.

But how to avoid it? There is certainly no way to isolate our codebase entirely from the framework, but we can minimize it by separating the concerns of our classes (or functions for those who prefer not to use Object Orientation). It requires a good feeling of what are the possible reasons to change a given component. For example: If a given class contains some of our business rules and could also be affected by a change in our framework, we have two possible actors requiring changes to the class. It is advisable to separate it into two different parts: one handling the framework and another one just concerned with the business rules.

When I’m working on long term projects, no matter the technology stacks, some responsibilities are always a good idea to separate since the beginning of the project. It is not a comprehensive list of how to separate the responsibilities of each class, but only an initial list of suggestions, no matter the nature of the project we’re working on. According to some specific business needs, more classes can rise here.

Entry points vs Controllers

What I am calling entry points here can have different names according to your favourite framework. It can be something like:

Django View

Celery Task

Flask route

Node/Express Middleware

Ruby on Rails Controller

The most important part of the entry point is that it must adhere to your framework requirements. The signature of the entry point method must follow some standards created by the framework.

Why separate them?

We can have several reasons to separate these concerns.

The first one is: we don’t want to tie our business rules with a given framework. If we decide to move to a different one, our business classes must be modified as well, adding additional risk to our migration. It also gives us two different reasons to change the same class: technical changes and business rules requests.

But only this reason can sound as over-engineering. Let’s be honest: we don’t change frameworks quite often. So we have other arguments for separating these concerns.

A second good reason is: we might want to start the same business rules from different places. For example: at first, the only way to trigger a given logic is through a user request. But down on the road, we might want to perform a periodic task to run the same logic.

A third scenario that becomes easier when we separate the concerns is the need to provide the same result in different formats. For example, a given result set can be returned as a JSON to be consumed by an app or a CSV file available for download. Our Controller must be the same, so it must not include any information about the output format. Once the Controller returns the result set, then we decide according to the parameters on the entry point, the proper format to return the records.

In any of these cases, this separation would lead us to:

Validate the input in the entry point;

Delegate the business processing to the Controller;

Receive the results from the Controller and prepare the response on the entry point.

If you are familiar with the SOLID principles, you’re probably thinking about the Single Responsibility Principle right now: we keep the entry point responsible only for the input validation, while the Controller process whatever was required.

In terms of architecture, it is crucial to ensure our Controller does not have any dependency on the entry points. The entry points are attached to our framework and we want to detach our Business Logic layer from any framework at any cost! Any parameter received by the entry points must become a parameter for the Controller, but removing any framework-specific type.

How to test each part?

Separate the entry point from the Controller can make our tests simpler as well.

Since the entry point is the layer responsible for input parameters validation, the unit test must focus on a thoroughly test to ensure the entry points reject any inappropriate input. During the entry points tests, we can mock the Controller since we don’t need to run it. All we need is to confirm the number of calls made to it and if the Controller receives the correct parameters. For example, an invalid input should not invoke the Controller, while we need to ensure that a valid request sends the correct parameters to the Controller.

The Controller does not need to include tests for invalid input since it was already tested on the entry points.

Controllers vs Response Adapters

Our controllers should be the most “pure” objects in our architecture. They should not be aware of any framework or technology. Imagine if we need to provide the same report in different formats. The business logic is the same, while the output can vary to JSON, CSV, HTML or any other format. So we need to ensure our controllers are not aware of any of these technologies and return only pure objects. Keeping the controllers away from technology allows us to reuse them in any desired output.

According to our needs in a given entry point, we might need to convert these pure objects into the proper response. For these cases, the Response Adapters are handy! They receive any response from the Controller and convert it into the format received by our entry point.

This separation also helps us to create more straightforward tests. The Controller now does not have to handle any technology requirements anymore. We must test the Response Adapter to get any meaningful response from the Controller and make sure it follows all the requirements for that specific format.

In some cases, we don’t need to provide the same response in different formats, but formatting the data received from the Controller to the output expected by the entry point is lengthy. In these cases, it is also advisable to keep the response adapter, to state clearly what is the business logic and what is only some data formatting.

So, after we separate our Controller from the entry point and the response adapters, our core business code does not need to handle input validation and data formatting anymore. Instead of a big monolith to handle our entry points, now we have different smaller pieces, like:

def my_entrypoint(request):

errors = _validate_request(request)

if errors:

return Response(errors, status=HTTP_400_BAD_REQUEST) data = my_controller(request.GET.get("my_param"))

return Response(_convert_my_data(data))

PS: The Python snippet above is not supposed to work on any specific framework. It only illustrates how concerns are separated into different parts.

Controllers vs Data Access Objects

At first glance, it may sound absurd to not access the database directly from the Controller. But let’s think a bit more about it. The benefits of this separation are not residing only in the remote possibility of migrating our database. But what if we decide to introduce caching? Should our business logic layer be aware of changes like this? Of course not! So all our Controller should know is that we’re invoking a method to retrieve records. But it doesn’t matter if the records are coming from MySQL or Redis. Our Controller should not have any reference to any framework (like Django models) or specific database (like MySQL, Mongo or Redis).

Down the road, when we decide to introduce caching for some requests, the Controller is not affected.

Another positive impact of having a data access layer is to make it easier to look for all queries sent to the database when we’re looking for some performance issues. The Data Access Objects (DAO) offer a centralized layer with all our statements, making it easier to identify the queries and possible index candidates.

Isolating the DAO from the Controller also allows us to mock the database responses, removing some latency from our tests. The controller tests run way faster!

Just a piece of advice here: it is a common issue to be tempted to move some logic to the Data Access Layer. Like some further processing right after retrieve data. Please, do not allow it! Data Access should only abstract the calls to the database. Anything else should belong to the Controller or any other layer you find useful.

While writing the tests, we need to ensure the DAO covers all the meaningful scenarios to access the database, while the Controller mocks the DAO and focus on how the Controller behaves according to some different responses. It is usually helpful to test scenarios with no data coming from and one/many records. If your business logic includes different behaviour according to some records types or statuses, you likely want to mock them as well.

Controllers vs Third-Party Clients

Controllers should not depend on external providers. And it includes any third-party library. There is a considerable chance that your system depends on an external provider. Maybe it is your cloud provider. Or maybe any system your application integrates with. You may be pretty happy with it. It is technically stable, evolves in a way that fits yours and your customers’ needs. But it might change for different reasons. Maybe technical or even commercial reasons. When it happens, how does it impact our system?

We can prepare for changes by limiting our exposure to third-party libraries. Instead of spreading them into our codebase, we can put them into a separate class and use factories to instantiate them. In some cases, our third-party client is just a proxy to the third-party library. But in other cases, some data adapter must be required, so instead of performing this conversion inside our Controller, we can move it to the new class.

While we are preparing our unit tests, the third party client strives to check every detail of our communication mocking the network requests. So we need to have a good understanding of the third-party API to mock them properly. It is also advised to include some integration tests that won’t mock the requests to ensure the third-party API is still returning what we’re expecting. It warns us about any undocumented change in the API and lets us know about the change before our customers! Since this integration test depends on network availability, it is not a good idea to include it in our CI/CD pipeline because it can fail more frequently. Maybe run it nightly or some times a day is good enough.

When unit testing the Controller, the client can be mocked to let us focus on the business rules. The mocks return the scenarios required to ensure the maximum useful coverage to our Controller.

Third-party Clients vs Adapters

When communicating with a 3rd party, we probably need to convert data from our app format into the third-party provider’s format, so it might be a good idea to move this conversion to a separate object. This separation cleans up our client to focus only on API communication.

Again, we can see benefits on testing, making it easier to test the conversion in detail without the need to mock any API or network request.

According to what we are doing, we might need conversion on both ways: on the parameters, we send to the third-party API and also convert the response into something provider agnostic to be passed to the Controller.

Ensuring loose coupling with factories — Python example

At this point, you might be convinced about the advantages of separating concerns and avoid dependencies. But how can we achieve that?

Two concepts come to our rescue here: Object-Oriented Interfaces and The Factory Design Pattern.

The interface lets us define the signatures of the methods for every class that wants to implement that interface. The only things defined are:

The method name

The parameters expected

The result of the method

No code needs to be reused here since each provider we want to integrate to have differences between each other.

Python offers a library named ABC: Abstract Base Classes. This library has two handy structures for us:

The ABC class — our interfaces must inherit from this class

The abstractmethod decorator — our methods must use this decorator to state it has to be implemented by classes inheriting the interface. We can’t use the client until a method implementation exists. If we try to use it, Python raises an exception.

Using these structures ensures any piece of code using a class that implements our interface has all the declared methods with the signature defined on the interface. We don’t care about the implementation. All we need is the public signature.

For example, let’s imagine an app that integrates with accounting systems. There are a lot of them in the market: Quickbooks Online, Netsuite, SAP and many others. If we need to send Bills from our system to these third-party systems, we could define an interface like:

from abc import ABC, abstractmethod class BillInterface(ABC): @abstractmethod

def sync_bills(bills_list) -> List[BillSyncStatus]:

pass

Any class that implements the BillInterface has to implement the sync_bills method. Each class has its implementation, but all of them return a list of BillSyncStatus objects. Maybe one uses a REST endpoint, another one has an old SOAP endpoint and a third one uses CSV. It doesn’t matter for the callers.

Now that we have different clients implementing the same interface, a new question arises: how to decide which one will be used? Certainly, there is an object or setting in our code base that holds the system we’re integrating with. So we can use this as a parameter to our factory method:

def make_bill_client(system):

if system == “Quickbooks”:

return QuickbooksBillClient()

elif system == “Netsuite”:

return NetsuiteBillClient()

elif system == “SAP”:

return SapBillClient()

Our Controller can now use the factory to get the third-party client instance even without any dependency on the external providers:

def my_bills_controller():

# find system to integrate with

# get the list of bills client = make_bill_client(system)

sync_response = client.sync_bills(bills_list)

All the Controller knows is that the client implements the interface. So it is safe to invoke the sync_bills method passing the list of bills as a parameter.

Ideally, the factory should be in a different module than the Controller to ensure the Controller not even imports anything related to the third-party system.

Wrapping up

It is common sense to have some minimum separation of concerns on our codebase. Most of us are currently using some framework to enforce some separation of concerns, like MVC or MVVM. It is a good starting point, but these frameworks set the barely minimum to allow easy maintenance.

Common additional layers to our controller-ish layer are:

Entry points — to separate our business logic from the framework

Clients — to separate business logic from the third-party libraries

Data Access Objects — to separate business logic from database access

Adapters — to separate complex or lengthy data transformation from any other concern

It may sound over-engineering. If you are working on your college assessment or just learning a new framework, you’re right. There is no need to go for this separation in small, short-lived projects. But for long term projects, it is something that certainly pays off.

As a positive side-effect, our tests run faster. Take a look at all these layers. What depends on external dependencies?

Clients can depend on external network APIs, but we must mock these calls to eliminate any network instability and latency. To make sure nothing changes on the third-party end and break our system, we should create some integration tests and run them separately from our unit tests. Maybe run two or three times a day can be good enough here.

Data Access Objects depend on the database, and it is not a good idea to mock them. So we probably want to move these tests to our integration tests as well.

According to some specific business or technical needs, you may need even more classes to properly ensure your classes are small, simple and focused enough for good maintainability.

If you are still curious about it, I recommend some topics: