Package: fan-tools
About this packageHere are set of internal tools that are shared between different projects internally. Originally most tools related to testing, so they provide some base classes for various cases in testingNOTE: all our tools are intentially support only 3.8+ python.Some might work with other versions, but we're going to be free from all these crutches to backport things likeasync/awaitto lower versions, so if it works - fine, if not - feel free to send PR, but it isn't going to be merged all times.Testing helpersCaching decorators# cache async function that returns pydantic.BaseModelfromfan_tools.pythonimportcache_async@cache_async[type(dict)](fname,model,{})asyncdeffunc():returnmodel# cache sync function that returns json serializable responsefromfan_tools.pythonimportmemoize@memoizedeffunc(*args,**kwargs):returnjson.dumps({'args':args,'kwargs':kwargs,})ApiUrlsDefined infan_tools/testing/__init__.py. Required for defining nested urls with formatting.You can use it in fixtures, like:@pytest.fixture(scope='session')defapi(api_v_base):yieldApiUrls('{}/'.format(api_v_base),{'password_reset_request':'password/request/code/','password_reset':'password/reset/','user_review_list':'user/{user_id}/review/','user_review':'user/{user_id}/review/{review_id}/','wine_review':'wine/{wine_id}/review/','drink_review':'drink/{drink_id}/review/',})deftest_review_list(user,api):resp=user.get_json(api.user_review_list(user_id=user1.id),{'page_size':2})PropsMetaYou can find source infan_tools/testing/meta.py.For now it convert methods that are started withprop__into descriptors with cache.classA(metaclass=PropsMeta):defprop__conn(self):conn=SomeConnection()returnconnBecame:classA:@propertydefconn(self):ifnothasattr(self,'__conn'):setattr(self,'__conn',SomeConnection())returnself.__connThus it allows quite nice style of testing with lazy initialization. Like:classMyTest(TestCase,metaclass=PropsMeta):defprop__conn(self):returnpsycopg2.connect('')defprop__cursor(self):returnself.conn.cursor()deftest_simple_query(self):self.cursor.execute('select 1;')row=self.cursor.fetchone()assertrow[0]==1,'Row:{}'.format(row)Here you just get and useself.cursor, but automatically you get connection and cursor and cache they.This is just simple example, complex tests can use more deep relations in tests. And this approach is way more easier and faster than complexsetUpmethods.fan_tools.unix helpersBasic unix helpersrun - run command in shellsucc - wrapper aroundrunwith return code and stderr checkwait_socket - wait for socket awailable (eg. you can wait for postgresql withwait_socket('localhost', 5432)asucc - asynchronous version ofsuccfor use withawait. supports realtime loggingsource - acts similar to bash 'source' or '.' commands.cd - contextmanager to do something with temporarily changed directoryinterpolate_sysenvFormat string with system variables + defaults.PG_DEFAULTS={'PGDATABASE':'postgres','PGPORT':5432,'PGHOST':'localhost','PGUSER':'postgres','PGPASSWORD':'',}DSN=interpolate_sysenv('postgresql://{PGUSER}:{PGPASSWORD}@{PGHOST}:{PGPORT}/{PGDATABASE}',PG_DEFAULTS)fan_tools.fan_logging.JSFormatterEnable json output with additional fields, suitable for structured logging into ELK or similar solutions.Acceptsenv_varskey with environmental keys that should be included into log.# this example uses safe_logger as handler (pip install safe_logger)importloggingimportlogging.configLOGGING={'version':1,'disable_existing_loggers':True,'formatters':{'json':{'()':'fan_tools.fan_logging.JSFormatter','env_vars':['HOME'],},'standard':{'format':'%(asctime)s[%(levelname)s]%(name)s:%(message)s'},},'handlers':{'default':{'level':'DEBUG','class':'safe_logger.TimedRotatingFileHandlerSafe','filename':'test_json.log','when':'midnight','interval':1,'backupCount':30,'formatter':'json',},},'loggers':{'':{'handlers':['default'],'level':'DEBUG',},},}logging.config.dictConfig(LOGGING)log=logging.getLogger('TestLogger')log.debug('test debug')log.info('test info')log.warn('test warn')log.error('test error')fan_tools.mon_server.MetricsServerFastAPI based server that servers metrics in prometheus format.import uvicorn

from fan_tools.mon_server.certs import update_certs_loop

app = FastAPI()
mserver = MetricsServer(app)
mserver.add_task(update_certs_loop, hosts=['perfectlabel.io', 'robopickles.com'])
uvicorn.run(app, host='0.0.0.0', port=os.environ.get('MONITORING_PORT', 8000))fan_tools.backupThere are two backup helpers:fan_tools.backup.s3.S3backupandfan_tools.backup.gcloud.GCloudWe're assuming that backup script has access to backup execution and dump directory.Default setup includes support for docker container that access DB.By default script provides interface for monitoring (last backup date).fan_tools.backup.s3.S3backupprovides external script calledfan_s3_backupthat has accepts some configuration via environmental variables.ENABLE_BACKUP- you need enable this by setting to nonfalsevalue, default: falseBACKUP_DB_CONTAINER- container for backup command executionBACKUP_DB_SCRIPT- command for exectuion on db server from above. default:/create_backup.pyBACKUP_COMMAND- overrides all above-b/--bucket- to define bucket. default for s3: environmental variableAWS_BACKUP_BUCKETBACKUP_PREFIXor-p/--prefix- directory backup prefix, usually it is subfolder for dumps, default:backups/-d/--daemonize- should we run in daemonized modeMONITORING_PORT- port for listen when run in daemonized mode. default: 80S3 specific:AWS_BACKUP_KEYAWS_BACKUP_SECRETAWS_BACKUP_BUCKETfan_tools.drf.serializers.EnumSerializerAllow you to deserealize incoming strings intoEnumvalues.
You should addEnumSerializerinto your serializers by hand.fromenumimportIntEnumfromdjango.dbimportmodelsfromrest_frameworkimportserializersfromfan_tools.drf.serializersimportEnumSerializerclassMyEnum(IntEnum):one=1two=2classExampleModel(models.Model):value=models.IntegerField(choices=[(x.name,x.value)forxinMyEnum])classExampleSerializer(serializers.ModelSerializer):value=EnumSerializer(MyEnum)# this allows you to post value as: {'value': 'one'}Due toEnumandIntegerFieldrealizations you may useEnum.valuein querysetsExampleModel.objects.filter(value=MyEnum.two)fan_tools.django.log_requests.LoggerMiddlewareLoggerMiddleware will log request meta + raw post data into log.For django<1.10 please usefan_tools.django.log_requests.DeprecatedLoggerMiddlewarefan_tools.django.request_uniqDecorator adds a unique for each uwsgi request dict as first function
argument.
For tests mock_get_request_unique_cachefan_tools.django.call_once_on_commitMake function called only once on transaction commit. Here is examples
where functiondo_some_usefulwill be called only once after
transaction has been committed.classSomeModel(models.Model):name=IntegerField()@call_once_on_commitdefdo_some_useful():passdefhook(sender,instance,**kwargs):do_some_useful()models.signals.post_save.connect(hook,sender=SomeModel)withtransaction.atomic():some_model=SomeModel()some_model.name='One'some_model.save()some_model.name='Two'some_model.save()For tests with nested transactions (commit actually most times is not
called) it is useful to override behaviourcall_once_on_commitwhen decorated function executed right in place where it is called.
To do so mockon_commitfunction. Example pytest fixture:@pytest.fixture(scope='session', autouse=True)
def immediate_on_commit():
    def side_effect():
        return lambda f: f()

    with mock.patch('fan_tools.django.on_commit', side_effect=side_effect) as m:
        yield mfan_tools.django.fields.ChoicesEnumUsed for choices attribute for in model fieldclass FooBarEnum(ChoicesEnum):
    foo = 1
    bar = 2

class ExampleModel(models.Model):
    type = models.IntegerField(choices=FooBarEnum.get_choices())fan_tools.django.db.utils.set_word_similarity_thresholdAllow to set postgres trigram word similarity threshold for default django database connectionset_word_similarity_threshold(0.4)fan_tools.django.contrib.postgres.models.LTreeModelDjango Model containing postgres ltreeclass LTreeExampleModel(LTreeModel):fan_tools.django.contrib.postgres.fields.LTreeDescendantsLookup for postgres ltree descendantsLTreeExampleModel.objects.filter(path__descendants='root.level1')fan_tools.django.contrib.postgres.fields.LTreeNlevelLookup for postgres ltree by level depthLTreeExampleModel.objects.filter(path__nlevel=2)fan_tools.django.db.pgfields.SimilarityLookupPostgrestext %> textoperator# Add this import to models.py (file should be imported before lookup usage)
import fan_tools.django.db.pgfields  # noqa

Books.objects.filter(title__similar='Animal Farm')fan_tools.django.db.pgfields.WordSimilarityPostgrestext1 <<-> text2operator. It returns1 - word_similarity(text1, text2)from django.db.models import Value, F

similarity = WordSimilarity(Value('Animal Farm'), F('title'))
Books.objects.annotate(similarity=similarity)fan_tools.drf.filters.NumberInFilterDjango filter that match if integer is in the integers list separated by commaclass ExampleFilterSet(FilterSet):
    example_values = NumberInFilter(field_name='example_value', lookup_expr='in')fan_tools.django.mail.MailSend text and html emails using django templates.Mail(
    recipient_list=[user.email],
    template_name='user/emails/reset_password',
    context={
        'frontend_url': settings.FRONTEND_URL,
    },
).send()fan_tools.django.url.build_absolute_uriGet domain section of absolute url of current page using django request object.build_absolute_uri(request)fan_tools.drf.forms.use_formHelps to use power of serializers for simple APIs checks.fromrest_frameworkimportserializersfromrest_framework.decoratorsimportapi_viewfromfan_tools.drfimportuse_formclassSimpleForm(serializers.Serializer):test_int=serializers.IntegerField()test_str=serializers.CharField()@api_view(['GET'])@use_form(SimpleForm)defmy_api(data):print(f'Data:{data["test_int"]}and{data["test_str"]}')fan_tools.drf.pagination.ApiPageNumberPaginationAllow turn off pagination by specifying zero page_zize.REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'fan_tools.drf.pagination.ApiPageNumberPagination',
    ...
}fan_tools.rest_framework.renderers.ApiRendererPretty Django Rest Framework API renderer with error codes.REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': (
        'fan_tools.drf.renderers.ApiRenderer',
    },
    ...
}fan_tools.rest_framework.handlers.api_exception_handlerPretty Django Rest Framework API exception handler with error codes.REST_FRAMEWORK = {
    'EXCEPTION_HANDLER': 'fan_tools.drf.handlers.api_exception_handler',
    ...
}fan_tools.drf.asserts.assert_validation_errorHelper assert function to be used in tests to match the validation error codes.assert_validation_error(response, 'email', 'unique')fan_tools.aio_utils.DbRecordsProcessorWorkerAsyncio worker which wait for new records in postgres db table and process them.fan_tools.aio_utils.dict_query/sql_updateaiopg shortcutsfan_tools.python.execfileBackport of python's 2execfilefunction.Usage: execfile('path/to/file.py', globals(), locals())Returns: True if file exists and executed, False if file doesn't existfan_tools.doc_utils.fan_sphinxSphinx extensions to generate documentation for django restframework serializers and examples for http requests.In order to use them specify dependency for package installation:pip install fan_tools[doc_utils]Usage:# Add to Sphinx conf.py
extensions = [
    # ...
    'fan_tools.doc_utils.fan_sphinx.dyn_serializer',
    'fan_tools.doc_utils.fan_sphinx.http_log'
]Commandsfan_env_yamlConvert template yaml with substituion of%{ENV_NAME}strings to appropriate environment variables.Usage:fan_env_yaml src_file dst_filefan_ci_scriptHelper to run default CI pipeline. Defaults are set up forgiltab defaults. Includes stages:build docker image with temporary name (commit sha by default)run tests (optional)push branch (by default only for master and staging branches)push tag, if there are tagscache image with common namedelete image with temporary nameIt's optimized for parallel launches, so you need to use unique temporary name (--temp-name). We want keep our system clean if possible, so we'll delete this tag in the end. But we don't want to repeat basic steps over and over, so we will cache image with common cache name (--cache-name), it will remove previous cached image.fan_waitWait for socket awailable/not-awailable with timeout.# Wait until database port up for 180 seconds
fan_wait -t 180 postgres 5432

# Wait until nginx port down for 30 seconds
fan_wait -t 30 nginx 80run_filebeatchecks environmental variables-e KEY=VALUE -e KEY2=VALUE2converts yaml templatefan_env_yaml {TEMPLATE} /tmp/filebeat.ymlrun/usr/bin/filebeat /tmp/filebeat.ymlrun_filebeat -e CHECKME=VALUE path_to_templatedoc_serializeroutput rst with list of serializersgenerates documentation artifacts for serializersusage: doc_serializer [-h] [--rst] [--artifacts]

Parse serializers sources

optional arguments:
  -h, --help   show this help message and exit
  --rst        Output rst with serializers
  --artifacts  Write serializers artifactsimage_utils.TransposeSave rotated by exif tag images. Some browsers/applications don't respect this tag,
so it is easier to do that explicitly.classImage(models.Model):uploaded_by=models.ForeignKey(User,blank=True,null=True,on_delete=models.SET_NULL)image=models.ImageField(blank=True,upload_to=image_upload_to)thumb_image=models.ImageField(blank=True,upload_to=thumb_upload_to)full_url=models.CharField(blank=True,max_length=255)thumb_url=models.CharField(blank=True,max_length=255)created=models.DateTimeField(auto_now_add=True)updated=models.DateTimeField(auto_now=True)classImageSerializer(ModelSerializer):classMeta:model=Imagefields=['id','created','updated','full_url','thumb_url']classUploadImageView(views.GenericAPIView):permission_classes=[IsAuthenticated]defpost(self,request,*args,**kwargs):image_data=request.data['image']# Fix an image orientation based on exif and remove exif from the resulted image.transformed_image=Transpose().process(image_data)obj=Image.objects.create(uploaded_by=request.user,image=transformed_image)obj.full_url=obj.image.urlobj.save()s=ImageSerializer(instance=obj)returnResponse(s.data)fan_tools.metricsHelper to send metrics.Example for datadogUsually you want to setup some kind of notification for metric with nameerror_metric. It is sent bysend_error_metric.For DataDog your metric query will look like:sum:error_metric{service:prod*} by {error_type,service}.as_count()development# keep docker containertox-epy311-django40----keep-dbdjango_tests
tox-epy311-django40----keep-db--docker-skipdjango_tests