Package: ohmyfpg
Oh My Fast Postgres!ohmyfpgis a Postgres client library for Python that aims to return data as columns. This is often needed when working with numerical data. Usually this is achieved by taking the output of the client library and then convert it either intonumpyarrays orpandasdataframes. When dealing with a large amount of data, this conversion is not much performant.The goal of this library is to return data already asnumpyarrays without sacrificing performance. The section of "Performance comparison" goes more in-depth on this topic.In order to squeeze performance the underlying implementation is written in Rust. The Python layer on top is very thin.Whyohmyfpg?When working with Postgres at work we faced multiple times performance issues. Most of the times our reactions were along the lines of: "OMG", "F*****g PG", etc. Soohmyfpgis kinda a mix of the two, but where thefnow stands forfast.(To be fair, when we faced performance issues with Postgres was most of the times because of our inexperience with tuning the server configurations.)Installationpip install ohmyfpgQuickstartimport asyncio
import ohmyfpg

DSN = 'postgres://postgres:postgres@postgres:5432/postgres'
QUERY = 'SELECT * FROM performance_test'

async def main():
    conn = await ohmyfpg.connect(DSN)
    print(await conn.fetch(QUERY))

if __name__ == '__main__':
    asyncio.run(main())Performance comparisonThe image below compares the performance ofohmyfpgwithasyncpg. The 4 bars have the following meaning:ohmyfpg: plain fetch,asyncpg: plain fetch,ohmyfpg-pandas: plain fetch + conversion topandasDataframe,asyncpg: plain fetch + conversion topandasDataframe,See detailshere, especially how the conversion topandasDataframe has been implemented.The query is aSELECT *that has been run on a table with 1mln rows and the following schema:(id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8)It has been run inside docker with 8 CPU and 8GB of RAM allocated to the daemon on a MBP with 2.2 GHz 6-Core Intel Core i7 and 16GB 2400 MHz DDR4.Detailed summaryPlain fetch:--------------------------------------------------
ohmyfpg
avg: 1045.8ms
min: 898ms
p25: 960.75ms
median: 1041.5ms
p75: 1083.0ms
max: 1421ms
--------------------------------------------------
asyncpg
avg: 1194.9ms
min: 1037ms
p25: 1080.5ms
median: 1224.0ms
p75: 1259.75ms
max: 1567ms
--------------------------------------------------Plain fetch + conversion topandasDataframe:--------------------------------------------------
ohmyfpg-pandas
avg: 1212.3ms
min: 1131ms
p25: 1166.5ms
median: 1192.0ms
p75: 1220.75ms
max: 1724ms
--------------------------------------------------
asyncpg-pandas
avg: 4013.0333333333333ms
min: 3771ms
p25: 3841.25ms
median: 3912.5ms
p75: 4124.5ms
max: 4708ms
--------------------------------------------------LimitationsThis library is highly experimental and has many limitations:no support forNULLs with unpredictable outcome,no support for non-numerical types,limited support for authentication,no proper logging,etc.DevelopmentHow to run the performance comparisondocker compose build script
docker compose up -d postgres
docker compose exec -- postgres psql -U postgres

CREATE TABLE performance_test (id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8);
INSERT INTO performance_test (
    id,
    foo_bar_int2,
    foo_bar_int4,
    foo_bar_int8,
    foo_bar_float4,
    foo_bar_float8
) VALUES (
    generate_series(1, 1000000),
    trunc(random() * (2*32768) - 32768),
    trunc(random() * (2*2147483648) - 2147483648),
    trunc(random() * (2*9223372036854775808) - 9223372036854775808),
    trunc(random()),
    trunc(random())
);


docker compose up script
docker compose cp script:/usr/src/app/performance-comparison.png ./performanceHow to do basic benchmarkingdocker run -p 5432:5432 --name rust-postgres -e POSTGRES_PASSWORD=postgres -d postgres -c log_min_messages=DEBUG5Data preparation:CREATE TABLE performance_test (id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8);
INSERT INTO performance_test (
    id,
    foo_bar_int2,
    foo_bar_int4,
    foo_bar_int8,
    foo_bar_float4,
    foo_bar_float8
) VALUES (
    generate_series(1, 1000000),
    trunc(random() * (2*32768) - 32768),
    trunc(random() * (2*2147483648) - 2147483648),
    trunc(random() * (2*9223372036854775808) - 9223372036854775808),
    trunc(random()),
    trunc(random())
);maturin develop --release --manifest-path ohmyfpg/Cargo.toml
python python/examples/simple_query.pyRUST_BACKTRACE=1 cargo run -r -p ohmyfpg_core --example simple_queryHow to do basic profilingsudo CARGO_PROFILE_BENCH_DEBUG=true RUST_BACKTRACE=1 cargo flamegraph -p ohmyfpg_core --example simple_queryCARGO_PROFILE_BENCH_DEBUG=true RUST_BACKTRACE=1 cargo instruments --release -p ohmyfpg_core --example simple_query -t time