Package: returns
Make your functions return something meaningful, typed, and safe!FeaturesBrings functional programming to Python landProvides a bunch of primitives to write declarative business logicEnforces better architectureFully typed with annotations and checked withmypy,PEP561 compatibleAdds emulated Higher Kinded Types supportProvides type-safe interfaces to create your own data-types with enforced lawsHas a bunch of helpers for better compositionPythonic and pleasant to write and to read 🐍Support functions and coroutines, framework agnosticEasy to start: has lots of docs, tests, and tutorialsQuickstartright now!InstallationpipinstallreturnsYou can also installreturnswith the latest supportedmypyversion:pipinstallreturns[compatible-mypy]You would also need to configure ourmypyplugin:# In setup.cfg or mypy.ini:[mypy]plugins=returns.contrib.mypy.returns_pluginor:[tool.mypy]plugins=["returns.contrib.mypy.returns_plugin"]We also recommend to use the samemypysettingswe use.Make sure you know how to get started,check out our docs!Try our demo.ContentsMaybe containerthat allows you to writeNone-free codeRequiresContext containerthat allows you to use typed functional dependency injectionResult containerthat let's you to get rid of exceptionsIO containerandIOResultthat marks all impure operations and structures themFuture containerandFutureResultto work withasynccodeWrite your own container!You would still have all the features for your own types (including full existing code reuse and type-safety)Maybe containerNoneis called theworst mistake in the history of Computer Science.So, what can we do to check forNonein our programs?
You can use builtinOptionaltype
and write a lot ofif some is not None:conditions.
But,havingnullchecks here and there makes your code unreadable.user:Optional[User]discount_program:Optional['DiscountProgram']=NoneifuserisnotNone:balance=user.get_balance()ifbalanceisnotNone:credit=balance.credit_amount()ifcreditisnotNoneandcredit>0:discount_program=choose_discount(credit)Or you can useMaybecontainer!
It consists ofSomeandNothingtypes,
representing existing state and empty (instead ofNone) state respectively.fromtypingimportOptionalfromreturns.maybeimportMaybe,maybe@maybe# decorator to convert existing Optional[int] to Maybe[int]defbad_function()->Optional[int]:...maybe_number:Maybe[float]=bad_function().bind_optional(lambdanumber:number/2,)# => Maybe will return Some[float] only if there's a non-None value#    Otherwise, will return NothingYou can be sure that.bind_optional()method won't be called forNothing.
Forget aboutNone-related errors forever!We can also bind aOptional-returning function over a container.
To achieve this, we are going to use.bind_optionalmethod.And here's how your initial refactored code will look:user:Optional[User]# Type hint here is optional, it only helps the reader here:discount_program:Maybe['DiscountProgram']=Maybe.from_optional(user,).bind_optional(# This won't be called if `user is None`lambdareal_user:real_user.get_balance(),).bind_optional(# This won't be called if `real_user.get_balance()` is Nonelambdabalance:balance.credit_amount(),).bind_optional(# And so on!lambdacredit:choose_discount(credit)ifcredit>0elseNone,)Much better, isn't it?RequiresContext containerMany developers do use some kind of dependency injection in Python.
And usually it is based on the idea
that there's some kind of a container and assembly process.Functional approach is much simpler!Imagine that you have adjangobased game, where you award users with points for each guessed letter in a word (unguessed letters are marked as'.'):fromdjango.httpimportHttpRequest,HttpResponsefromwords_app.logicimportcalculate_pointsdefview(request:HttpRequest)->HttpResponse:user_word:str=request.POST['word']# just an examplepoints=calculate_points(user_word)...# later you show the result to user somehow# Somewhere in your `words_app/logic.py`:defcalculate_points(word:str)->int:guessed_letters_count=len([letterforletterinwordifletter!='.'])return_award_points_for_letters(guessed_letters_count)def_award_points_for_letters(guessed:int)->int:return0ifguessed<5elseguessed# minimum 6 points possible!Awesome! It works, users are happy, your logic is pure and awesome.
But, later you decide to make the game more fun:
let's make the minimal accountable letters threshold
configurable for an extra challenge.You can just do it directly:def_award_points_for_letters(guessed:int,threshold:int)->int:return0ifguessed<thresholdelseguessedThe problem is that_award_points_for_lettersis deeply nested.
And then you have to passthresholdthrough the whole callstack,
includingcalculate_pointsand all other functions that might be on the way.
All of them will have to acceptthresholdas a parameter!
This is not useful at all!
Large code bases will struggle a lot from this change.Ok, you can directly usedjango.settings(or similar)
in your_award_points_for_lettersfunction.
Andruin your pure logic with framework specific details. That's ugly!Or you can useRequiresContextcontainer. Let's see how our code changes:fromdjango.confimportsettingsfromdjango.httpimportHttpRequest,HttpResponsefromwords_app.logicimportcalculate_pointsdefview(request:HttpRequest)->HttpResponse:user_word:str=request.POST['word']# just an examplepoints=calculate_points(user_words)(settings)# passing the dependencies...# later you show the result to user somehow# Somewhere in your `words_app/logic.py`:fromtypingimportProtocolfromreturns.contextimportRequiresContextclass_Deps(Protocol):# we rely on abstractions, not direct values or typesWORD_THRESHOLD:intdefcalculate_points(word:str)->RequiresContext[int,_Deps]:guessed_letters_count=len([letterforletterinwordifletter!='.'])return_award_points_for_letters(guessed_letters_count)def_award_points_for_letters(guessed:int)->RequiresContext[int,_Deps]:returnRequiresContext(lambdadeps:0ifguessed<deps.WORD_THRESHOLDelseguessed,)And now you can pass your dependencies in a really direct and explicit way.
And have the type-safety to check what you pass to cover your back.
Check outRequiresContextdocs for more. There you will learn how to make'.'also configurable.We also haveRequiresContextResultfor context-related operations that might fail. And alsoRequiresContextIOResultandRequiresContextFutureResult.Result containerPlease, make sure that you are also aware ofRailway Oriented Programming.Straight-forward approachConsider this code that you can find inanypythonproject.importrequestsdeffetch_user_profile(user_id:int)->'UserProfile':"""Fetches UserProfile dict from foreign API."""response=requests.get('/api/users/{0}'.format(user_id))response.raise_for_status()returnresponse.json()Seems legit, does it not?
It also seems like a pretty straightforward code to test.
All you need is to mockrequests.getto return the structure you need.But, there are hidden problems in this tiny code sample
that are almost impossible to spot at the first glance.Hidden problemsLet's have a look at the exact same code,
but with the all hidden problems explained.importrequestsdeffetch_user_profile(user_id:int)->'UserProfile':"""Fetches UserProfile dict from foreign API."""response=requests.get('/api/users/{0}'.format(user_id))# What if we try to find user that does not exist?# Or network will go down? Or the server will return 500?# In this case the next line will fail with an exception.# We need to handle all possible errors in this function# and do not return corrupt data to consumers.response.raise_for_status()# What if we have received invalid JSON?# Next line will raise an exception!returnresponse.json()Now, all (probably all?) problems are clear.
How can we be sure that this function will be safe
to use inside our complex business logic?We really cannot be sure!
We will have to createlotsoftryandexceptcases
just to catch the expected exceptions. Our code will become complex and unreadable with all this mess!Or we can go with the top levelexcept Exception:case
to catch literally everything.
And this way we would end up with catching unwanted ones.
This approach can hide serious problems from us for a long time.Pipe exampleimportrequestsfromreturns.resultimportResult,safefromreturns.pipelineimportflowfromreturns.pointfreeimportbinddeffetch_user_profile(user_id:int)->Result['UserProfile',Exception]:"""Fetches `UserProfile` TypedDict from foreign API."""returnflow(user_id,_make_request,bind(_parse_json),)@safedef_make_request(user_id:int)->requests.Response:# TODO: we are not yet done with this example, read more about `IO`:response=requests.get('/api/users/{0}'.format(user_id))response.raise_for_status()returnresponse@safedef_parse_json(response:requests.Response)->'UserProfile':returnresponse.json()Now we have a clean and a safe and declarative way
to express our business needs:We start from making a request, that might fail at any moment,Then parsing the response if the request was successful,And then return the result.Now, instead of returning regular values
we return values wrapped inside a special container
thanks to the@safedecorator. It will returnSuccess[YourType] or Failure[Exception].
And will never throw exception at us!We also useflowandbindfunctions for handy and declarative composition.This way we can be sure that our code won't break in
random places due to some implicit exception.
Now we control all parts and are prepared for the explicit errors.We are not yet done with this example,
let's continue to improve it in the next chapter.IO containerLet's look at our example from another angle.
All its functions look like regular ones:
it is impossible to tell whether they arepureor impure from the first sight.It leads to a very important consequence:we start to mix pure and impure code together.
We should not do that!When these two concepts are mixed
we suffer really bad when testing or reusing it.
Almost everything should be pure by default.
And we should explicitly mark impure parts of the program.That's why we have createdIOcontainer
to mark impure functions that never fail.These impure functions userandom, current datetime, environment, or console:importrandomimportdatetimeasdtfromreturns.ioimportIOdefget_random_number()->IO[int]:# or use `@impure` decoratorreturnIO(random.randint(1,10))# isn't pure, because randomnow:Callable[[],IO[dt.datetime]]=impure(dt.datetime.now)@impuredefreturn_and_show_next_number(previous:int)->int:next_number=previous+1print(next_number)# isn't pure, because does IOreturnnext_numberNow we can clearly see which functions are pure and which ones are impure.
This helps us a lot in building large applications, unit testing you code,
and composing business logic together.Troublesome IOAs it was already said, we useIOwhen we handle functions that do not fail.What if our function can fail and is impure?
Likerequests.get()we had earlier in our example.Then we have to use a specialIOResulttype instead of a regularResult.
Let's find the difference:Our_parse_jsonfunction always returns
the same result (hopefully) for the same input:
you can either parse validjsonor fail on invalid one.
That's why we return pureResult, there's noIOinsideOur_make_requestfunction is impure and can fail.
Try to send two similar requests with and without internet connection.
The result will be different for the same input.
That's why we must useIOResulthere: it can fail and hasIOSo, in order to fulfill our requirement and separate pure code from impure one,
we have to refactor our example.Explicit IOLet's make ourIOexplicit!importrequestsfromreturns.ioimportIOResult,impure_safefromreturns.resultimportsafefromreturns.pipelineimportflowfromreturns.pointfreeimportbind_resultdeffetch_user_profile(user_id:int)->IOResult['UserProfile',Exception]:"""Fetches `UserProfile` TypedDict from foreign API."""returnflow(user_id,_make_request,# before: def (Response) -> UserProfile# after safe: def (Response) -> ResultE[UserProfile]# after bind_result: def (IOResultE[Response]) -> IOResultE[UserProfile]bind_result(_parse_json),)@impure_safedef_make_request(user_id:int)->requests.Response:response=requests.get('/api/users/{0}'.format(user_id))response.raise_for_status()returnresponse@safedef_parse_json(response:requests.Response)->'UserProfile':returnresponse.json()And later we can useunsafe_perform_iosomewhere at the top level of our program to get the pure (or "real") value.As a result of this refactoring session, we know everything about our code:Which parts can fail,Which parts are impure,How to compose them in a smart, readable, and typesafe manner.Future containerThere are several issues withasynccode in Python:You cannot callasyncfunction from a sync oneAny unexpectedly thrown exception can ruin your whole event loopUgly composition with lots ofawaitstatementsFutureandFutureResultcontainers solve these issues!Mixing sync and async codeThe main feature ofFutureis that it allows to run async code
while maintaining sync context. Let's see an example.Let's say we have two functions,
thefirstone returns a number and thesecondone increments it:asyncdeffirst()->int:return1defsecond():# How can we call `first()` from here?returnfirst()+1# Boom! Don't do this. We illustrate a problem here.If we try to just runfirst(), we will just create an unawaited coroutine.
It won't return the value we want.But, if we would try to runawait first(),
then we would need to changesecondto beasync.
And sometimes it is not possible for various reasons.However, withFuturewe can "pretend" to call async code from sync code:fromreturns.futureimportFuturedefsecond()->Future[int]:returnFuture(first()).map(lambdanum:num+1)Without touching ourfirstasync function
or makingsecondasync we have achieved our goal.
Now, our async value is incremented inside a sync function.However,Futurestill requires to be executed inside a proper eventloop:importanyio# or asyncio, or any other lib# We can then pass our `Future` to any library: asyncio, trio, curio.# And use any event loop: regular, uvloop, even a custom one, etcassertanyio.run(second().awaitable)==2As you can seeFutureallows you
to work with async functions from a sync context.
And to mix these two realms together.
Use rawFuturefor operations that cannot fail or raise exceptions.
Pretty much the same logic we had with ourIOcontainer.Async code without exceptionsWe have already covered howResultworks
for both pure and impure code.
The main idea is: we don't raise exceptions, we return them.
It isespeciallycritical in async code,
because a single exception can ruin
all our coroutines running in a single eventloop.We have a handy combination ofFutureandResultcontainers:FutureResult.
Again, this is exactly likeIOResult, but for impure async code.
Use it when yourFuturemight have problems:
like HTTP requests or filesystem operations.You can easily turn any wild throwing coroutine into a calmFutureResult:importanyiofromreturns.futureimportfuture_safefromreturns.ioimportIOFailure@future_safeasyncdefraising():raiseValueError('Not so fast!')ioresult=anyio.run(raising.awaitable)# all `Future`s return IO containersassertioresult==IOFailure(ValueError('Not so fast!'))# TrueUsingFutureResultwill keep your code safe from exceptions.
You can alwaysawaitor execute inside an eventloop anyFutureResultto get syncIOResultinstance to work with it in a sync manner.Better async compositionPreviously, you had to do quite a lot ofawaiting while writingasynccode:asyncdeffetch_user(user_id:int)->'User':...asyncdefget_user_permissions(user:'User')->'Permissions':...asyncdefensure_allowed(permissions:'Permissions')->bool:...asyncdefmain(user_id:int)->bool:# Also, don't forget to handle all possible errors with `try / except`!user=awaitfetch_user(user_id)# We will await each time we use a coro!permissions=awaitget_user_permissions(user)returnawaitensure_allowed(permissions)Some people are ok with it, but some people don't like this imperative style.
The problem is that there was no choice.But now, you can do the same thing in functional style!
With the help ofFutureandFutureResultcontainers:importanyiofromreturns.futureimportFutureResultE,future_safefromreturns.ioimportIOSuccess,IOFailure@future_safeasyncdeffetch_user(user_id:int)->'User':...@future_safeasyncdefget_user_permissions(user:'User')->'Permissions':...@future_safeasyncdefensure_allowed(permissions:'Permissions')->bool:...defmain(user_id:int)->FutureResultE[bool]:# We can now turn `main` into a sync function, it does not `await` at all.# We also don't care about exceptions anymore, they are already handled.returnfetch_user(user_id).bind(get_user_permissions).bind(ensure_allowed)correct_user_id:int# has required permissionsbanned_user_id:int# does not have required permissionswrong_user_id:int# does not exist# We can have correct business results:assertanyio.run(main(correct_user_id).awaitable)==IOSuccess(True)assertanyio.run(main(banned_user_id).awaitable)==IOSuccess(False)# Or we can have errors along the way:assertanyio.run(main(wrong_user_id).awaitable)==IOFailure(UserDoesNotExistError(...),)Or even something really fancy:fromreturns.pointfreeimportbindfromreturns.pipelineimportflowdefmain(user_id:int)->FutureResultE[bool]:returnflow(fetch_user(user_id),bind(get_user_permissions),bind(ensure_allowed),)Later we can also refactor our logical functions to be sync
and to returnFutureResult.Lovely, isn't it?More!Want more?Go to the docs!Or read these articles:Python exceptions considered an anti-patternEnforcing Single Responsibility Principle in PythonTyped functional Dependency Injection in PythonHow Async Should Have BeenHigher Kinded Types in PythonMake Tests a Part of Your AppDo you have an article to submit? Feel free to open a pull request!