Package: isitbullshit
isitbullshitis small and funny library which is intended to be used like lightweight schema verification for JSONs
but basically it could be used as a schema validator for every generic Python structure: dict, list, tuple etc. It is
written to be pretty much Pythonic in a good sense: easy to use and very clean syntax but powerful enough to clean
your needs. But mostly for verification of incoming JSONs. Actually it is really stable and I am using it in several
production projects, this is an excerpt because I really got tired of reinventing the wheel.Yes, this is a wheel reinvention also but probably you will like it. Let me show the code.An exampleOkay, let’s say you are doing some backend for the library and you have to process JSONs like this:{"model":"book_collection","pk":318,"fields":{"books":[{"model":"book","pk":18,"fields":{"title":"Jane Eyre","author":"Charlotte Brontë","isbn":{"10":"0142437204","13":"978-0142437209"},"rate":null,"language":"English","type":"paperback","tags":["Penguin Classics","Classics","Favorites"],"published":{"publisher":"Penguin Books","date":{"day":24,"month":4,"year":2003}}}},{"model":"book","pk":18,"fields":{"title":"The Great Gatsby","author":"F.Scott Fitzgerald","isbn":{"10":"185326041X","13":"978-1853260414"},"language":"English","type":"paperback","finished":true,"rate":4,"tags":["Wordsworth Classics","Classics","Favorites"],"published":{"publisher":"Wordsworth Editions Ltd","date":{"day":1,"month":5,"year":1992}}}}]}}You’ve got an idea, right? Pretty common and rather simple. Let’s compose a schema and verify it.fromjsonimportloadsfromisitbullshitimportisitbullshit,OrSkippeddefrate_validator(value):ifnot(1<=int(value)<=5):raiseValueError("Value{}has to be from 1 till 5".format(value))data=loads(request)schema={"model":str,"pk":int,"fields":{"books":[{"model":str,"pk":int,"fields":{"title":str,"author":str,"isbn":{"10":str,"13":str},"language":str,"type":("paperback","kindle"),"finished":OrSkipped(True),"rate":(rate_validator,None),"tags":[str],"published":{"publisher":str,"date":OrSkipped({"day":int,"month":int,"year":int})}}}]}}ifisitbullshit(data,schema):raiseError400("Incoming request is not valid")process(data)Pretty straightforward. Let me explain what is going on here.Basic conceptsisitbullshit was created to be used with JSONs and actively uses the fact that JSON perfectly matches to Python
internal data structures. Basic rule here: if elements are equal then they should be validated without any problems.So if you have a code like>>>suspicious={..."foo":1,..."bar":2...}then>>>printisitbullshit(suspicious,suspicious)FalseKeep this in mind.If elements are equal then no additional validation steps have to be used. Otherwise it tries to match types and do
some explicit assertions.So there are some rules.Value validationValue validation is pretty straighforward: if values are the same or they are equal to each other (operation=)
then validation has to be passed. So the rule is: ifisor=works, then matching is successful.>>>printisitbullshit(1,1)False>>>printisitbullshit(1.0,1.0)False>>>printisitbullshit(1.0,decimal.Decimal("1.0"))False>>>printisitbullshit(None,None)False>>>obj=object()>>>printisitbullshit(obj,obj)FalseType validationIf value validation is not passed then type validation is performed. The idea is:1is1, right? But you will
be satisfied if you know that1isintas well, right?So>>>printisitbullshit(1,int)False>>>printisitbullshit(1.0,float)False>>>printisitbullshit(decimal.Decimal("1.0"),decimal.Decimal)False>>>obj=object()>>>printisitbullshit(obj,object)FalseCustom validationLet’s get back to an example. Have you mentioned that we haverate_validatorfunction there? It is custom validator.It works pretty simple: you define custom callable (function, lambda, class, etc) andisitbullshitgives it your
value. If no exception is raised than we consider the value as successfully validated. So in our example if aratefield is not in (1, 5) interval or not integer then exception will be raised.Custom validators are used mostly in cases if you have to check a content or do not so shallow verifications.But there is only one pitfall you may face with:custom validators have to be a functions. Basically, this is an
obligatory rule and there are several reasons. Let’s checkout the code:>>>printisitbullshit(1,str)What do you expect to have as result? I guessTruebecause integer is not an instance of the string type, right?
But wait a minute, in Python 2:>>>type(str)<type'type'>and in Python 3>>>type(str)<class'type'>So they are types! They have the same type as, let’s say,Exceptionorobject, right? But validation rules
have to be consistent so I am trying to keep absolutely the same behaviour to have it clean and predictable.What is the story? Here is the story:>>>printstr(1)'1'So to avoid such situations whenisitbullshit(1, str) == Falseit is better to use functions. Functions are the most
reasonable agreement I see here. So if you want to verify MongoDB’s ObjectId’s do the following:>>>printisitbullshit(1,lambdavalue:bson.ObjectId(value))True>>>printisitbullshit("507c7f79bcf86cd7994f6c0e",lambdavalue:bson.ObjectId(value))FalseIt brings some clutter but at least you will not hike in the minefield.OrSkipped validatorSometimes we live in a real world which sucks. Sometimes we have schemaless data (and it sucks of course) so some
fields from your requests are missed. Or you do not care.isitbullshithas 2 different fixes for
that:OrSkippedandWHATEVER.If you wrap a part of your validator inOrSkippedthan you mark that it is ok if this field would be absent.
Argument is a validator of course. And if field is in place, it will be validated as expected.>>>schema={..."foo":1,..."bar":OrSkipped(int),..."baz":OrSkipped(str)>>>}>>>printisitbullshit({"foo":1,"bar":1},schema)False>>>printisitbullshit({"foo":1,"bar":"str"},schema)True>>>printisitbullshit({"foo":1,"bar":1,"baz":1},schema)True>>>printisitbullshit({"foo":1,"bar":1,"baz":"str"},schema)FalseSo if we miss any field, it is ok. Unless it is presented and validator-argument point us to a bullshit.OrSkippedhas to be used only with dictionary field validation. You can put it anywhere but then it has no special
meaning, just an object.By the way, type validation rule is still here:itisbullshit(something, something) == Falseanyway so the following
code is valid (and it is reasonable, right?)>>>schema={..."foo":1,..."bar":OrSkipped(int),..."baz":OrSkipped(str)>>>}>>>isitbullshit(schema,schema)False>>>stripped_schema=dict((k,v)fork,vinschema.iteritems()ifk!="baz")>>>isitbullshit(stripped_schema,schema)False>>>isitbullshit(schema,stripped_schema)FalseGuess why.WHATEVER validatorWHATEVERis a mark that you do not care what value is. It could be anything, nobody cares.>>>schema={..."foo":1,..."bar":WHATEVER>>>}>>>printisitbullshit({"foo":1,"bar":1},schema)False>>>printisitbullshit({"foo":1,"bar":"str"},schema)False>>>printisitbullshit({"foo":1,"bar":object()},schema)False>>>printisitbullshit({"foo":1,"bar":os.path},schema)False>>>printisitbullshit({"foo":1,"bar":[1,2,3]},schema)FalseSee? We do not care about a value of abar.WHATEVERcould be used with any type.Dict validationYou’ve already seen adictvalidation so let me repeat your assumptions: yes, we match values with the same keys.
But there is only one pitfall: if suspicious element has more fields than schema, then validation is ok also.It has it’s own meaning: we can put only those keys and fields we actually care about. Our software later will work
only with this subset so why should we care about the rest of rubbish?So, an example again:>>>schema={..."foo":1,..."bar":str>>>}>>>printisitbullshit({"foo":1,"bar":"st"},schema)False>>>printisitbullshit({"foo":1,"bar":"str","baz":1},schema)False>>>printisitbullshit({"foo":1,"bar":"str","baz":object()},schema)FalseAs you can see, we did not mention anybazin an element but validation still passed.List validationList validation is pretty simple: we define one validator and it will be matched to any list element.>>>printisitbullshit([1,2,3],[int])False>>>printisitbullshit([1,2,3],[str])True>>>printisitbullshit([1,2,"3"],[int])TrueIn the last example,"3"is not an integer so validation fails.How could we manage situations when we have heterogeneous elements? We have to use tuples.And please remember thatisitbullshit(something, something) == False.Tuple validationTuple validation is pretty easy to understand if you consider it as an OR condition. We define several validators
and and the value has to match at least one of them. So>>>printisitbullshit(1,(str,dict))True>>>printisitbullshit(1,(str,int))False1is notstrbut it isint.Now let’s try to fix an example in the previous chapter.>>>printisitbullshit([1,2,"3"],[int])True>>>printisitbullshit([1,2,"3"],[(int,str)])FalseAnd again, do not forget about a rule of thumb:isitbullshit(something, something) == False.raise_for_problem functionThis package also provides you with another method to validate,raise_for_problemactually this is a core method
which raises an exception on a problem.isitbullshitallows you to get an idea what is happening in both Python2 and
Python3, let’s check an example.>>>try:...raise_for_problem({"foo":"1","bar":{"baz":2}},{"foo":"1","bar":{"baz":str}})...exceptItIsBullshitErroraserr:...printerr{'foo':'1','bar':{'baz':2}}:{'baz':2}:2:Schememismatch<type'str'>Quite clear and nice. If you want just to extract a pure message lines, iterateItIsBullshitErrorand
you are good.IsItBullshitMixin mixinisitbullshitalso supplied withIsItBullshitMixinwhich is intended to be mixed withunittest.TestCase. It
allows you to use 2 additional methods:assertBullshitassertNotBullshitGuess what they do.fromunittestimportTestCasefromisitbullshitimportIsItBullshitMixinclassBullshitTestCase(IsIsBullshitMixin,TestCase):deftest_bullshit(self):self.assertBullshit(1,None)deftest_not_bullshit(self):self.assertNotBullshit(1,int)