Package: ucall
UCallJSON Remote Procedure Calls LibraryUp to 100x Faster than FastAPIMost modern networking is built either on slow and ambiguous REST APIs or unnecessarily complex gRPC.
FastAPI, for example, looks very approachable.
We aim to be equally or even simpler to use.FastAPIUCallpipinstallfastapiuvicornpipinstallucallfromfastapiimportFastAPIimportuvicornserver=FastAPI()@server.get('/sum')defsum(a:int,b:int):returna+buvicorn.run(...)fromucall.posiximportServer# from ucall.uring import Server on 5.19+server=Server()@serverdefsum(a:int,b:int):returna+bserver.run()It takes over a millisecond to handle a trivial FastAPI call on a recent 8-core CPU.
In that time, light could have traveled 300 km through optics to the neighboring city or country, in my case.
How does UCall compare to FastAPI and gRPC?Setup🔁ServerLatency w 1 clientThroughput w 32 clientsFast API over REST❌🐍1'203 μs3'184 rpsFast API over WebSocket✅🐍86 μs11'356 rps ¹gRPC ²✅🐍164 μs9'849 rpsUCall with POSIX❌C62 μs79'000 rpsUCall with io_uring✅🐍40 μs210'000 rpsUCall with io_uring✅C22 μs231'000 rpsTable legendAll benchmarks were conducted on AWS on general purpose instances withUbuntu 22.10 AMI.
It is the first major AMI to come withLinux Kernel 5.19, featuring much widerio_uringsupport for networking operations.
These specific numbers were obtained onc7g.metalbeefy instances with Graviton 3 chips.The 🔁 column marks, if the TCP/IP connection is being reused during subsequent requests.The "server" column defines the programming language, in which the server was implemented.The "latency" column report the amount of time between sending a request and receiving a response. μ stands for micro, μs subsequently means microseconds.The "throughput" column reports the number of Requests Per Second when querying the same server application from multiple client processes running on the same machine.¹ FastAPI couldn't process concurrent requests with WebSockets.² We tried generating C++ backends with gRPC, but its numbers, suspiciously, weren't better. There is also an async gRPC option, that wasn't tried.How is that possible?!How can a tiny pet-project with just a couple thousand lines of code compete with two of the most established networking libraries?UCall stands on the shoulders of Giants:io_uringfor interrupt-less IO.io_uring_prep_read_fixedon 5.1+.io_uring_prep_accept_directon 5.19+.io_uring_register_files_sparseon 5.19+.IORING_SETUP_COOP_TASKRUNoptional on 5.19+.IORING_SETUP_SINGLE_ISSUERoptional on 6.0+.SIMD-accelerated parsers with manual memory control.simdjsonto parse JSON faster than gRPC can unpackProtoBuf.Turbo-Base64to decode binary values from aBase64form.picohttpparserto navigate HTTP headers.You have already seen the latency of the round trip..., the throughput in requests per second..., want to see the bandwidth?
Try yourself!@serverdefecho(data:bytes):returndataMore Functionality than FastAPIFastAPI supports native type, while UCall supportsnumpy.ndarray,PIL.Imageand other custom types.
This comes handy when you build real applications or want to deploy Multi-Modal AI, like we do withUForm.fromucall.rich_posiximportServerimportufromserver=Server()model=uform.get_model('unum-cloud/uform-vl-multilingual')@serverdefvectorize(description:str,photo:PIL.Image.Image)->numpy.ndarray:image=model.preprocess_image(photo)tokens=model.preprocess_text(description)joint_embedding=model.encode_multimodal(image=image,text=tokens)returnjoint_embedding.cpu().detach().numpy()We also have our own optionalClientclass that helps with those custom types.fromucall.clientimportClientclient=Client()# Explicit JSON-RPC call:response=client({'method':'vectorize','params':{'description':description,'image':image,},'jsonrpc':'2.0','id':100,})# Or the same with syntactic sugar:response=client.vectorize(description=description,image=image)CLI likecURLAside from the PythonClient, we provide an easy-to-use Command Line Interface, which comes withpip install ucall.
It allow you to call a remote server, upload files, with direct support for images and NumPy arrays.
Translating previous example into a Bash script, to call the server on the same machine:ucallvectorizedescription='Product description'-iimage=./local/path.pngTo address a remote server:ucallvectorizedescription='Product description'-iimage=./local/path.png--uri0.0.0.0-p8545To print the docs, useucall -h:usage: ucall [-h] [--uri URI] [--port PORT] [-f [FILE ...]] [-i [IMAGE ...]] [--positional [POSITIONAL ...]] method [kwargs ...]

UCall Client CLI

positional arguments:
  method                method name
  kwargs                method arguments

options:
  -h, --help            show this help message and exit
  --uri URI             server uri
  --port PORT           server port
  -f [FILE ...], --file [FILE ...]
                        method positional arguments
  -i [IMAGE ...], --image [IMAGE ...]
                        method positional arguments
  --positional [POSITIONAL ...]
                        method positional argumentsYou can also explicitly annotate types, to distinguish integers, floats, and strings, to avoid ambiguity.ucall auth id=256
ucall auth id:int=256
ucall auth id:str=256Free Tier ThroughputWe will leave bandwidth measurements to enthusiasts, but will share some more numbers.
The general logic is that you can't squeeze high performance from Free-Tier machines.
Currently AWS provides following options:t2.microandt4g.small, on older Intel and newer Graviton 2 chips.
This library is so fast, that it doesn't need more than 1 core, so you can run a fast server even on a tiny Free-Tier server!Setup🔁ServerClientst2.microt4g.smallFast API over REST❌🐍1328 rps424 rpsFast API over WebSocket✅🐍11'504 rps3'051 rpsgRPC✅🐍11'169 rps1'974 rpsUCall with POSIX❌C11'082 rps2'438 rpsUCall with io_uring✅C1-5'864 rpsUCall with POSIX❌C323'399 rps39'877 rpsUCall with io_uring✅C32-88'455 rpsIn this case, every server was bombarded by requests from 1 or a fleet of 32 other instances in the same availability zone.
If you want to reproduce those benchmarks, check out thesumexamples on GitHub.Quick StartFor Python:pipinstallucallFor CMake projects:include(FetchContent)FetchContent_Declare(ucallGIT_REPOSITORYhttps://github.com/unum-cloud/ucallGIT_SHALLOWTRUE)FetchContent_MakeAvailable(ucall)include_directories(${ucall_SOURCE_DIR}/include)The C usage example is mouthful compared to Python.
We wanted to make it as lightweight as possible and to allow optional arguments without dynamic allocations and named lookups.
So unlike the Python layer, we expect the user to manually extract the arguments from the call context withucall_param_named_i64(), and its siblings.#include<cstdio.h>#include<ucall/ucall.h>staticvoidsum(ucall_call_tcall,ucall_callback_tag_t){int64_ta{},b{};charprinted_sum[256]{};boolgot_a=ucall_param_named_i64(call,"a",0,&a);boolgot_b=ucall_param_named_i64(call,"b",0,&b);if(!got_a||!got_b)returnucall_call_reply_error_invalid_params(call);intlen=snprintf(printed_sum,256,"%ll",a+b);ucall_call_reply_content(call,printed_sum,len);}intmain(intargc,char**argv){ucall_server_tserver{};ucall_config_tconfig{};ucall_init(&config,&server);ucall_add_procedure(server,"sum",&sum,NULL);ucall_take_calls(server,0);ucall_free(server);return0;}RoadmapBatch RequestsJSON-RPC over raw TCP socketsJSON-RPC over TCP with HTTPConcurrent sessionsNumPyarrayand Pillow serializationHTTPSsupportBatch-capable endpoints for MLZero-ETL relay callsIntegrating withUKVWebSockets for web interfacesAF_XDP and UDP-based analogs on LinuxWant to affect the roadmap and request a feature? Join the discussions on Discord.Why JSON-RPC?Transport independent: UDP, TCP, bring what you want.Application layer is optional: use HTTP or not.Unlike REST APIs, there is just one way to pass arguments.