Package: tipsi-tools
About this packageHere are set of internal tools that are shared between different projects internally. Originally most tools related to testing, so they provide some base classes for various cases in testingNOTE: all our tools are intentially support only 3.5+ python.Some might work with other versions, but we're going to be free from all these crutches to backport things likeasync/awaitto lower versions, so if it works - fine, if not - feel free to send PR, but it isn't going to be merged all times.Testing helpersApiUrlsDefined intipsi_tools/testing/__init__.py. Required for defining nested urls with formatting.You can use it in fixtures, like:@pytest.fixture(scope='session')defapi(api_v_base):yieldApiUrls('{}/'.format(api_v_base),{'password_reset_request':'password/request/code/','password_reset':'password/reset/','user_review_list':'user/{user_id}/review/','user_review':'user/{user_id}/review/{review_id}/','wine_review':'wine/{wine_id}/review/','drink_review':'drink/{drink_id}/review/',})deftest_review_list(user,api):resp=user.get_json(api.user_review_list(user_id=user1.id),{'page_size':2})PropsMetaYou can find source intipsi_tools/testing/meta.py.For now it convert methods that are started withprop__into descriptors with cache.classA(metaclass=PropsMeta):defprop__conn(self):conn=SomeConnection()returnconnBecame:classA:@propertydefconn(self):ifnothasattr(self,'__conn'):setattr(self,'__conn',SomeConnection())returnself.__connThus it allows quite nice style of testing with lazy initialization. Like:classMyTest(TestCase,metaclass=PropsMeta):defprop__conn(self):returnpsycopg2.connect('')defprop__cursor(self):returnself.conn.cursor()deftest_simple_query(self):self.cursor.execute('select 1;')row=self.cursor.fetchone()assertrow[0]==1,'Row:{}'.format(row)Here you just get and useself.cursor, but automatically you get connection and cursor and cache they.This is just simple example, complex tests can use more deep relations in tests. And this approach is way more easier and faster than complexsetUpmethods.AIOTestCaseNOTE: we're highly suggest to use pytest alongside with existing async tests pluginsBase for asyncronous test cases, you can use it as drop-in replacement for pre-existent tests to be able:write asyncronous test methodswrite asyncronoussetUpandtearDownmethodsuse asyncronous function inassertRaisesclassExampleCase(AIOTestCase):asyncsetUp(self):awaitasync_setup()asynctearDown(self):awaitasync_teardown()asyncdivision(self):1/0asynctest_example(self):awaitself.assertRaises(ZeroDivisionError,self.async_division)tipsi_tools.unix helpersBasic unix helpersrun - run command in shellsucc - wrapper aroundrunwith return code and stderr checkwait_socket - wait for socket awailable (eg. you can wait for postgresql withwait_socket('localhost', 5432)asucc - asynchronous version ofsuccfor use withawait. supports realtime loggingsource - acts similar to bash 'source' or '.' commands.cd - contextmanager to do something with temporarily changed directoryinterpolate_sysenvFormat string with system variables + defaults.PG_DEFAULTS={'PGDATABASE':'postgres','PGPORT':5432,'PGHOST':'localhost','PGUSER':'postgres','PGPASSWORD':'',}DSN=interpolate_sysenv('postgresql://{PGUSER}:{PGPASSWORD}@{PGHOST}:{PGPORT}/{PGDATABASE}',PG_DEFAULTS)tipsi_tools.tipsi_logging.JSFormatterEnable json output with additional fields, suitable for structured logging into ELK or similar solutions.Acceptsenv_varskey with environmental keys that should be included into log.# this example uses safe_logger as handler (pip install safe_logger)importloggingimportlogging.configLOGGING={'version':1,'disable_existing_loggers':True,'formatters':{'json':{'()':'tipsi_tools.tipsi_logging.JSFormatter','env_vars':['HOME'],},'standard':{'format':'%(asctime)s[%(levelname)s]%(name)s:%(message)s'},},'handlers':{'default':{'level':'DEBUG','class':'safe_logger.TimedRotatingFileHandlerSafe','filename':'test_json.log','when':'midnight','interval':1,'backupCount':30,'formatter':'json',},},'loggers':{'':{'handlers':['default'],'level':'DEBUG',},},}logging.config.dictConfig(LOGGING)log=logging.getLogger('TestLogger')log.debug('test debug')log.info('test info')log.warn('test warn')log.error('test error')tipsi_tools.mon_server.MetricsServerSanic based server that servers metrics in prometheus format.from sanic import Sanic

from tipsi_tools.mon_server import MetricsServer
from tipsi_tools.mon_server.certs import update_certs_loop

app = Sanic()
mserver = MetricsServer(app, status_metric='running{server="localhost"}')
mserver.add_task(update_certs_loop, hosts=['gettipsi.com', 'proofnetwork.io'])
app.run(host='0.0.0.0', port=8000)tipsi_tools.drf.serializers.EnumSerializerAllow you to deserealize incoming strings intoEnumvalues.
You should addEnumSerializerinto your serializers by hand.fromenumimportIntEnumfromdjango.dbimportmodelsfromrest_frameworkimportserializersfromtipsi_tools.drf.serializersimportEnumSerializerclassMyEnum(IntEnum):one=1two=2classExampleModel(models.Model):value=models.IntegerField(choices=[(x.name,x.value)forxinMyEnum])classExampleSerializer(serializers.ModelSerializer):value=EnumSerializer(MyEnum)# this allows you to post value as: {'value': 'one'}Due toEnumandIntegerFieldrealizations you may useEnum.valuein querysetsExampleModel.objects.filter(value=MyEnum.two)tipsi_tools.django.log_requests.LoggerMiddlewareLoggerMiddleware will log request meta + raw post data into log.For django<1.10 please usetipsi_tools.django.log_requests.DeprecatedLoggerMiddlewaretipsi_tools.django.request_uniqDecorator adds a unique for each uwsgi request dict as first function
argument.
For tests mock_get_request_unique_cachetipsi_tools.django.call_once_on_commitMake function called only once on transaction commit. Here is examples
where functiondo_some_usefulwill be called only once after
transaction has been committed.classSomeModel(models.Model):name=IntegerField()@call_once_on_commitdefdo_some_useful():passdefhook(sender,instance,**kwargs):do_some_useful()models.signals.post_save.connect(hook,sender=SomeModel)withtransaction.atomic():some_model=SomeModel()some_model.name='One'some_model.save()some_model.name='Two'some_model.save()For tests with nested transactions (commit actually most times is not
called) it is useful to override behaviourcall_once_on_commitwhen decorated function executed right in place where it is called.
To do so mockon_commitfunction. Example pytest fixture:@pytest.fixture(scope='session', autouse=True)
def immediate_on_commit():
    def side_effect():
        return lambda f: f()

    with mock.patch('tipsi_tools.django.on_commit', side_effect=side_effect) as m:
        yield mtipsi_tools.django.fields.ChoicesEnumUsed for choices attribute for in model fieldclass FooBarEnum(ChoicesEnum):
    foo = 1
    bar = 2

class ExampleModel(models.Model):
    type = models.IntegerField(choices=FooBarEnum.get_choices())tipsi_tools.django.db.utils.set_word_similarity_thresholdAllow to set postgres trigram word similarity threshold for default django database connectionset_word_similarity_threshold(0.4)tipsi_tools.django.contrib.postgres.models.LTreeModelDjango Model containing postgres ltreeclass LTreeExampleModel(LTreeModel):tipsi_tools.django.contrib.postgres.fields.LTreeDescendantsLookup for postgres ltree descendantsLTreeExampleModel.objects.filter(path__descendants='root.level1')tipsi_tools.django.contrib.postgres.fields.LTreeNlevelLookup for postgres ltree by level depthLTreeExampleModel.objects.filter(path__nlevel=2)tipsi_tools.django.db.pgfields.SimilarityLookupPostgrestext %> textoperator# Add this import to models.py (file should be imported before lookup usage)
import tipsi_tools.django.db.pgfields  # noqa

Books.objects.filter(title__similar='Animal Farm')tipsi_tools.django.db.pgfields.WordSimilarityPostgrestext1 <<-> text2operator. It returns1 - word_similarity(text1, text2)from django.db.models import Value, F

similarity = WordSimilarity(Value('Animal Farm'), F('title'))
Books.objects.annotate(similarity=similarity)tipsi_tools.drf.filters.NumberInFilterDjango filter that match if integer is in the integers list separated by commaclass ExampleFilterSet(FilterSet):
    example_values = NumberInFilter(field_name='example_value', lookup_expr='in')tipsi_tools.django.mail.MailSend text and html emails using django templates.Mail(
    recipient_list=[user.email],
    template_name='user/emails/reset_password',
    context={
        'frontend_url': settings.FRONTEND_URL,
    },
).send()tipsi_tools.django.url.build_absolute_uriGet domain section of absolute url of current page using django request object.build_absolute_uri(request)tipsi_tools.drf.forms.use_formHelps to use power of serializers for simple APIs checks.fromrest_frameworkimportserializersfromrest_framework.decoratorsimportapi_viewfromtipsi_tools.drfimportuse_formclassSimpleForm(serializers.Serializer):test_int=serializers.IntegerField()test_str=serializers.CharField()@api_view(['GET'])@use_form(SimpleForm)defmy_api(data):print(f'Data:{data["test_int"]}and{data["test_str"]}')tipsi_tools.drf.pagination.ApiPageNumberPaginationAllow turn off pagination by specifying zero page_zize.REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'tipsi_tools.drf.pagination.ApiPageNumberPagination',
    ...
}tipsi_tools.rest_framework.renderers.ApiRendererPretty Django Rest Framework API renderer with error codes.REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': (
        'tipsi_tools.drf.renderers.ApiRenderer',
    },
    ...
}tipsi_tools.rest_framework.handlers.api_exception_handlerPretty Django Rest Framework API exception handler with error codes.REST_FRAMEWORK = {
    'EXCEPTION_HANDLER': 'tipsi_tools.drf.handlers.api_exception_handler',
    ...
}tipsi_tools.drf.asserts.assert_validation_errorHelper assert function to be used in tests to match the validation error codes.assert_validation_error(response, 'email', 'unique')tipsi_tools.aio_utils.DbRecordsProcessorWorkerAsyncio worker which wait for new records in postgres db table and process them.tipsi_tools.aio_utils.dict_query/sql_updateaiopg shortcutstipsi_tools.python.execfileBackport of python's 2execfilefunction.Usage: execfile('path/to/file.py', globals(), locals())Returns: True if file exists and executed, False if file doesn't existtipsi_tools.doc_utils.tipsi_sphinxSphinx extensions to generate documentation for django restframework serializers and examples for http requests.In order to use them specify dependency for package installation:pip install tipsi_tools[doc_utils]Usage:# Add to Sphinx conf.py
extensions = [
    # ...
    'tipsi_tools.doc_utils.tipsi_sphinx.dyn_serializer',
    'tipsi_tools.doc_utils.tipsi_sphinx.http_log'
]Commandstipsi_env_yamlConvert template yaml with substituion of%{ENV_NAME}strings to appropriate environment variables.Usage:tipsi_env_yaml src_file dst_filetipsi_ci_scriptHelper to run default CI pipeline. Defaults are set up forgiltab defaults. Includes stages:build docker image with temporary name (commit sha by default)run tests (optional)push branch (by default only for master and staging branches)push tag, if there are tagscache image with common namedelete image with temporary nameIt's optimized for parallel launches, so you need to use unique temporary name (--temp-name). We want keep our system clean if possible, so we'll delete this tag in the end. But we don't want to repeat basic steps over and over, so we will cache image with common cache name (--cache-name), it will remove previous cached image.tipsi_waitWait for socket awailable/not-awailable with timeout.# Wait until database port up for 180 seconds
tipsi_wait -t 180 postgres 5432

# Wait until nginx port down for 30 seconds
tipsi_wait -t 30 nginx 80run_filebeatchecks environmental variables-e KEY=VALUE -e KEY2=VALUE2converts yaml templatetipsi_env_yaml {TEMPLATE} /tmp/filebeat.ymlrun/usr/bin/filebeat /tmp/filebeat.ymlrun_filebeat -e CHECKME=VALUE path_to_templatedoc_serializeroutput rst with list of serializersgenerates documentation artifacts for serializersusage: doc_serializer [-h] [--rst] [--artifacts]

Parse serializers sources

optional arguments:
  -h, --help   show this help message and exit
  --rst        Output rst with serializers
  --artifacts  Write serializers artifacts