Django : Finding ORM N+1 Problem in TestCode
Tags: Testcode, Django, Nplusone

Web Frameworks that use ORM may have the same N+1 Problem.

In this post, i will share how to find and fix it in testcode in advance

Refer to the link for explanation of N+1 Problem.

(https://medium.com/@hckcksrl/django-n-1-problem-d986b93f5d3e)

For each framework, packages that detect n+1 problems are already shared by great developers.

Django has the https://github.com/jmcarp/nplusone package, and we want to use it to solve the n+1 problem in the test code

First, install the package. For installation instructions, refer to the package readme.

The example code below is a sample code that writes a simple rest api based on resources.

There are two models, Article and Comment, which is a one-to-many relationship. There are multiple comments in one article.

Model

from django.db import models # Create your models here.

class Article(models.Model):

# user = models.ForeignKey(User, on_delete=models.CASCADE)

title = models.CharField(max_length=144)

subtitle = models.CharField(max_length=144, blank=True)

content = models.TextField()

created_at = models.DateTimeField(auto_now_add=True) def __str__(self):

return '[{}] {}'.format(self.title, self.subtitle)

class Comment(models.Model):

article = models.ForeignKey(to=Article, related_name="comments", on_delete=models.CASCADE)

content = models.TextField()

Article — Comment : one-to-many

View

from django.shortcuts import render # Create your views here.

from requests import Response

from rest_framework import viewsets

from .serializers import ArticleSerializer, CommentSerializer

from .models import Article, Comment

from rest_framework import permissions class ArticleView(viewsets.ModelViewSet):

queryset = Article.objects.all()

serializer_class = ArticleSerializer

permission_classes = () def list(self, request, *args, **kwargs):

return super().list(request, *args, **kwargs) def get_serializer(self, *args, **kwargs):

return super().get_serializer(*args, **kwargs) def get_serializer_context(self):

context = super(ArticleView, self).get_serializer_context() return context

class CommentView(viewsets.ModelViewSet): serializer_class = CommentSerializer

permission_classes = () def get_queryset(self):

return Comment.objects.filter(article=self.kwargs['article_pk']) def perform_create(self, serializer):

serializer.save()

It returns the resource in rest format.

Serializer

from rest_framework import serializers

from .models import Article, Comment

from django.contrib.auth.models import User

class CommentSerializer(serializers.ModelSerializer):

class Meta:

model = Comment

fields = (

'id',

'article',

'content'

)

class ArticleSerializer(serializers.ModelSerializer):

comments = CommentSerializer(many=True, read_only=True)

class Meta:

model = Article

fields = (

'id',

'title',

'subtitle',

'content',

'created_at',

'comments',

)

read_only_fields = ('created_at',)

ArticleSerializer brings multiple comments models (Important, it is not prefetched.)

Urls

from django.urls import path, include

from rest_framework_nested import routers from .views import ArticleView, CommentView router = routers.SimpleRouter()

router.register(r'articles', ArticleView, basename='articles') articles_router = routers.NestedSimpleRouter(router, r'articles', lookup='article')

articles_router.register(r'comments', CommentView, basename='article-comments')

urlpatterns = [ path('', include(router.urls)),

path('', include(articles_router.urls))

]

TestCode

from unittest import mock import pytest

from django.conf import settings

from django.test import Client from api.models import Article, Comment

@pytest.fixture

def logger(monkeypatch):

mock_logger = mock.Mock()

monkeypatch.setattr(settings, 'NPLUSONE_LOGGER', mock_logger)

return mock_logger def check_nplusone_problem(logger):

if len(logger.log.call_args_list) != 0: # include erros.

args = logger.log.call_args[0]

assert ("Potential n+1 query detected on" in args[1]) is False # if you need prefetch

assert ("Potential unnecessary eager load detected on" in args[1]) is False # if you not need prefetch assert not logger.log.called # In normal case, it should not be called def create_mock_article() -> Article:

"""

Make Article Mock Data.

:return:

"""

# Given

name: str = "TEST_title"

description: str = "description" # When

article: Article = Article()

article.title = name

article.subtitle = 'subtitle'

article.content = description

article.save() comment: Comment = Comment()

comment.content = "AAA"

comment.article = article

comment.save() return article

@pytest.mark.django_db

def test_route_article_GET(logger) -> None:

"""

You can test the url method GET of View

:return:

""" # Given

create_mock_article()

create_mock_article()

create_mock_article() client = Client()

# When

response = client.get('/api/articles/')

# Then

print(response.data)

check_nplusone_problem(logger)

assert response.status_code == 200

Test result -> Failed

-> The reason it fails is that ArticleSerializer() refers to multiple model comments, but it is not prefetching.

Correcting the failure case

Add Comment model to prefetch to ArticleView’s QuerySet

class ArticleView(viewsets.ModelViewSet):

queryset = Article.objects.prefetch('comments').all()

Test result -> Success

=================================================================================== 1 passed in 7.85s ===================================================================================

Evaluation

Even if the development team does not prefetch, there is no accumulated data, so performance issues are not immediately raised, but as data accumulates in operation, a bottleneck occurs in areas that require prefetching. This can be corrected and resolved in the test code stage in advance

PS : i am not good at English. Please tell me the wrong grammar and typo