Package: pandas-maxminddb
Pandas MaxmindProvides fast and convenient geolocation bindings forPandasDataframes. Usesnumpyndarray's internally to speed it up compared to naively
applying function per column. Based on
themaxminddb-rust.FeaturesSupports bothMMAPand in-memory implementationsSupports parallelism (useful for very big datasets)Comes with pre-built wheels, no need to install and maintain external C-library to get (better than) C-performanceInstallationMinimal supported Python is 3.8pip install pandas_maxminddbThe preferred way is to use precompiled binary wheel, as this requires no toolchain and is
fastest.If you want to build from source any
platformRust has targetfor is
supported.Pre-built wheelsThe wheels are built against followingnumpyandpandasdistributions:If you're on Windows / macOS / Linux there is no need to do anything extra.If you use ARMv7 (RaspberryPi and such)
usePiWheels--extra-index-url=https://www.piwheels.org/simple,
installlibatlas-base-devfor numpy.If you usemusl-based distro like Alpine
useAlpine-wheels--extra-index-url https://alpine-wheels.github.io/index, installlibstdc++for pandas.Refer to thebuild workflowfor details.Pywin x86win x64macOS x86_64macOS AArch64linux x86_64linux i686linux AArch64linux ARMv7musl linux x86_643.8✅✅✅✅✅✅✅🚫✅3.9✅✅✅✅✅✅✅✅🚫3.10🚫✅✅✅✅✅🚫🚫✅UsageBy importingpandas_maxminddbyou add Pandasgeoextension which allows you to add columns
in-place. This example uses context manager for reader lifetime:importpandasaspdfrompandas_maxminddbimportopen_databaseips=pd.DataFrame(data={'ip':["75.63.106.74","132.206.246.203","94.226.237.31","128.119.189.49","2.30.253.245"]})withopen_database('./GeoLite.mmdb/GeoLite2-City.mmdb')asreader:ips.geo.geolocate('ip',reader,['country','city','state','postcode'])ipsipcitypostcodestatecountry075.63.106.74Houston77070TXUS1132.206.246.203MontrealH3AQCCA294.226.237.31Kapellen2950VLGBE3128.119.189.49Northampton01060MAUS42.30.253.245LondonSW15ENGGBWithout context managerYou can also instantiate reader yourself, eg:importpandasaspdfrompandas_maxminddbimportReaderMem,ReaderMmapreader=ReaderMem('./GeoLite.mmdb/GeoLite2-City.mmdb')ips=pd.DataFrame(data={'ip':["75.63.106.74","132.206.246.203","94.226.237.31","128.119.189.49","2.30.253.245"]})ips.geo.geolocate('ip',reader,['country','city','state','postcode'])ipsParallelismIf dataset is big enough, and you have extra cores you might benefit from using them. Currently onlyReaderMemis supported:importpandasaspdfrompandas_maxminddbimportReaderMemreader=ReaderMem('./GeoLite.mmdb/GeoLite2-City.mmdb')ips=pd.DataFrame(data={'ip':["75.63.106.74","132.206.246.203","94.226.237.31","128.119.189.49","2.30.253.245"]})ips.geo.geolocate('ip',reader,['country','city','state','postcode'],parallel=True)ipsBenchmarksTested on M1 Max with 1024 chunk size on 100k dataset, refer tobenchmarkName (time in ms)MinMaxMeanStdDevMedianIQROutliersOPSRoundsIterationstest_benchmark_pandas_parallel_mem_maxminddb52.7588 (1.0)57.4206 (1.0)54.0573 (1.0)1.1782 (1.15)53.8497 (1.0)1.4194 (1.09)4;118.4989 (1.0)201test_benchmark_pandas_mmap_maxminddb240.0050 (4.55)244.3257 (4.26)242.2177 (4.48)1.9017 (1.85)243.1021 (4.51)3.2122 (2.46)2;04.1285 (0.22)51test_benchmark_pandas_mem_maxminddb241.4630 (4.58)244.2553 (4.25)242.8391 (4.49)1.0288 (1.0)242.7672 (4.51)1.3064 (1.0)2;04.1180 (0.22)51test_benchmark_c_maxminddb1,010.6569 (19.16)1,055.1080 (18.38)1,021.3691 (18.89)18.9273 (18.40)1,013.3819 (18.82)12.9544 (9.92)1;10.9791 (0.05)51test_benchmark_python_maxminddb9,021.2686 (170.99)9,188.7629 (160.03)9,071.0055 (167.80)70.0512 (68.09)9,039.7811 (167.87)84.7766 (64.89)1;00.1102 (0.01)51ExtendingDue to Dataframe columns being flat arrays and geolocation data coming in a hierarchical format you
might need to provide more mappings to serve your particular use-case. In order to do that follow
Development section to setup your environment and then:Add column name to thegeo_column.rsAdd column mapping to thegeolocate.rsDevelopmentSetting up environmentgit clone --recurse-submodules git@github.com:andrusha/pandas-maxminddb.gitPYTHON_CONFIGURE_OPTS="--enable-shared" asdf installPYTHON_CONFIGURE_OPTS="--enable-shared" python -m venv .venvsource .venv/bin/activatepip install noxnox -s testPYTHONPATH=.venv/lib/python3.8/site-packages cargo test --no-default-featureslibmaxminddbIn order to runnox -s benchproperly you would
needlibmaxminddbinstalled as
permaxminddbinstructions prior to
installing Python package, so that C-extension could be benchmarked properly.On macOS this would require following:brew instal libmaxminddbPATH="/opt/homebrew/Cellar/libmaxminddb/1.7.1/bin:$PATH" LDFLAGS="-L/opt/homebrew/Cellar/libmaxminddb/1.7.1/lib" CPPFLAGS="-I/opt/homebrew/Cellar/libmaxminddb/1.7.1/include" pip install maxminddb --force-reinstall --verbose --no-cache-dir