Saravanan666/pypi-org huggingface dataset:
Package: census2020
Helper functions for Census 2020 dataEvery decade the US Census Bureau releases data from itsdecennial census.
However, the files they provide are quite complicated. And while they provideSASandR, they don't provide any help for Python.This package provides some convenience functions for playing around with all of this
Census data in Python.RequirementsWe require Python 3.7.1 or above. This package does usepyarrowto make manipulating these large data sets easier. However, on some systems, you may encounter installation troubles. If you do, feel free to file an issue!To install the package, simply runpipinstallcensus2020UsageGetting the dataTo use this package, you should first download the Census data. We've included a simple
CLI for you to grab all of the data and preprocess it:census2020pull-all--outputdataHeredatais a folder into which all the processed data will be dumped. WARNING: It
totals about 1.4GB after it's processed.If for some reason CLI doesn't work, you can pull it by hand as follows:frompathlibimportPathimportpyarrow.parquetaspqimportusfromcensus2020importdownloaderoutput_dir=Path("data")forstateinsorted(set(us.STATES)|{us.states.DC}):print(f"Downloading{state.name}...")table=downloader.get_state(state.abbr)pq.write_table(table,output_dir/f"{state.abbr.lower()}.parquet")print(f"Done with{state.name}")Historical dataThere's a good chance you're downloading this data to compare it to historical data.
If so, you can download the PL94 data from the 2010 Census similarly to above. Just
runcensus2020pull-all--outputdata2010--year2010Or if the CLI doesn't work for you, change the loop above to read:table=downloader.get_state(state.abbr,year=2010)Reading the dataReading in all the data into memory can be a bit of a difficult task, so we have
provided some interfaces topyarrow's filtering features to help.For example, suppose you wanted the total population of people who identify as both
White and Asian in all Census Tracts in Kentucky, Indiana, and Ohio. Assuming you
have downloaded all the data, you can run the following code:fromcensus2020importreadersfromcensus2020.constantsimportSummaryLeveldf=readers.read_filtered_dataset("data",states=["KY","IN","OH"],levels=SummaryLevel.STATE_COUNTY_TRACT,columns="P0010013",).to_pandas()Here"data"is the location to which you downloaded the Census data, which can be
either the 2020 or 2010 data.Each ofstates,columns, andlevelscan be either singular values or lists of
values. If no value is specified, then all states, columns, and levels available
will be returned.CodebookMore detail on the information in these files is available from theCensus Bureau. In particular, a summary of the fields meanings is available inthis Excel file, reporduced in this repo asfield_names.xlsx.LicenseMIT
Package: census2dbf
Converts .csv files downloaded from theCensus
Factfinderinto the legacyDBFformat, which is useful for
GIS applications. Census2dbf is a standalone command line tool with no
requirements.Installpip install census2dbfUsageOn the shell:$ census2dbf path/to/census.csv --output=/path/of/saved.dbf
path/of/saved.dbfThe--outputargument is optional. If it’s omitted, a file with the
same name but a .dbf extension will be saved in the same folder. E.g.census-file.csvwill be converted to a new file namedcensus-file.dbfOptions--dd: Data dictionaryUse the--ddoption to generate a data dictionary from the headers
of the census file. The data dictionary will be saved to the same folder
as the output dbf, with a similar name.Although Census data files usually come with a data dictionary,census2dbfwill sometimes rewrite fieldnames to avoid duplicates.
The data dictionary will match the$ census2dbf --dd tests/data/ACS_13_5YR_S0802_with_ann.csv
tests/data/ACS_13_5YR_S0802_with_ann.dbf
tests/data/ACS_13_5YR_S0802_with_ann-data-dictionary.txt
$ head tests/data/ACS_13_5YR_S0802_with_ann-data-dictionary.txt
Data Dictionary
Automatically extracted from the header of tests/data/ACS_13_5YR_S0802_with_ann.csv
2015-03-18 15:09
GEO.id  Id
...--include-cols: Filter out columnsPass the--include-colsor-ioption a comma-delimited list of
column names to output only those columns. This option is
case-insensitive.$ census2dbf --include-cols geo.id,GEO.id,GEO.id2,HD02_S21 path/to/census.csv
Package: census-api
Census API PackageThis package is a wrapper for querying the US Census API.InstallationYou can installcensus_apiusing pip.pipinstallcensus_apiBranchesThemasterbranch contains the current state ofcensus_apias it is hosted on PyPI. Thedevbranch contains the next version ofcensus_apiin development.Do not commit directly to themasterbranch.Make commits in thedevbranch and then PR to themasterbranch before uploading to PyPI.Changelogv0.0.2:Fixed__init__.pyimport errorv0.0.1:Initial release!
Package: census-area
This Python library extends the Sunlight Foundation’sCensus API Wrapperto allow querying Census tracts, block groups, and blocks by Census place, as well as by  arbitrary geographies.SetupGet the library and its dependencies usingpip:pip install census_areaUsageCheck out the documentation on Read The Docs.TeamJean Cochrane, DataMadeForest Gregg, DataMadeHannah Cushman Garland, DataMadeErrors and bugsIf something is not behaving intuitively, it is a bug and should be reported.
Report it here by creating an issue:https://github.com/datamade/census_area/issuesHelp us fix the problem as quickly as possible by followingMozilla’s guidelines for reporting bugs.Patches and pull requestsYour patches are welcome. Here’s our suggested workflow:Fork the project.Make your feature addition or bug fix.Send us a pull request with a description of your work. Bonus points for topic branches!Copyright and attributionCopyright (c) 2022 DataMade. Released under theMIT License.
Package: census-cb
census-cbcensus-cbis a wrapper for downloading and processing Cartography Boundaries from the US Census Bureau. It stands for Census Cartographic Boundaries; apparently that's confusing (but, like most things, it made sense at the time).The goal ofcensus-cbis to provide an easy way to get GIS information created by the United States Census Bureau. There's a wide variety of cartographic files availbable for download, such as state and county lines, voting district, Tribal subdivisions, and more!Entity InformationThere are a designated set of entities available for download from the census bureau. Each year of available data has a specific set of available entities, and the shape of these entities may change from year to year. You can find a full listpdf of available entities for downloadon the US Census Bureau site.InstallationInstall usingpippip install census-cbUsage# Create a Boundary File for the desired entity
bf = BoundaryFile(2020, 'us', 'state', '500k')

# Create a processor for downloading and unpacking the data; this processor will return a GeoDataFrame
cbfp = CBFProcessor('gdf')

# Downloads and extracts the Boundary File.
cbfp.process_data(bf)ExamplesDownload and display the state boudnaries from the US Census Bureau.# Create a Boundary File for the US State Lines
state_boundary_file = BoundaryFile(2020, 'us', 'state', '500k')

# Create a processor that returns a GeoDataFrame
cbfp = CBFProcessor('gdf')

# Get the Data
state_lines = cbfp.process_data(state_boundary_file)

# See the Data
state_lines.plot()
state_lines.head()geometrySTATEFPSTATENSAFFGEOIDGEOIDSTUSPSNAMELSADALANDAWATER0MULTIPOLYGON (((144.64538 13.23627, 144.64716 ...66018027050400000US6666GUGuam005435558479343374531MULTIPOLYGON (((-94.71830 29.72885, -94.71721 ...48017798010400000US4848TXTexas00676680588914189793522302MULTIPOLYGON (((-86.95617 45.35549, -86.95463 ...55017798060400000US5555WIWisconsin00140292246684293437216503MULTIPOLYGON (((-71.28802 41.64558, -71.28647 ...44012198350400000US4444RIRhode Island00267775921913236911294MULTIPOLYGON (((-72.03683 41.24984, -72.03496 ...36017797960400000US3636NYNew York0012204952086119256750161To Doprovide some better functionality for editing Entity Information after a BoundaryFile object is declared.
Package: censuscoding
censuscodingcensuscodingis a self-contained tool for determining the Census
blockgroup for a street address. It is developed byResearch Improving People's Lives (RIPL).InstallationDue to outstanding bugs in dependencies introduced in Python 3.10.0,censuscodingrequires Python >=3.7 and <3.10.To install from PyPI usingpip:pip install censuscodingTo install adevelopment versionfrom the current directory:pip install -e .RunningThere is a single command line script included,censuscoding.censuscodingsupports the following arguments:--inputor-i- path to an input CSV file containing columnsrecord_id,zip_code, andaddress--outputor-o- path to output CSV file containingrecord_id,zip_code,blkgrp--record_id- field name corresponding to the record ID (default: 'record_id')--zip_code- field name corresponding to the zip code (default: 'zip_code')--address- field name corresponding to the street address with street name and number (default: 'address')ContributorsMark HowisonDaniel MolitorLicensecensuscodingis freely available for non-commercial use under the license provided inLICENSE.
To inquiry about commercial use, please contactconnect@ripl.org.DataData included in thecensuscodingpackage were derived from the following sources:Rhode Island Geographic Information System E911 Site/Structure Address PointsVersione911Sites21r1.These data are designed and maintained for RI E 9-1-1 mapping software. They are not intended for any other purpose. This dataset is provided 'as is.â€™ The producer(s) of this dataset, contributors to this dataset, the Rhode Island Geographic Information System (RIGIS) consortium, the State of Rhode Island, and the University of Rhode Island do not make any warranties of any kind for this dataset, and are not liable for any loss or damage however and whenever caused by any use of this dataset. Please acknowledge both RIGIS and the primary producer(s) of this dataset in any derived products. Versions of the RIGIS logo suitable for both printed and web-based products are available athttps://www.rigis.org.US Census Bureau TIGER/Line ShapefilesVersion2020.
Package: census-consumer-complaint
How to setup airflowSet airflow directoryexport AIRFLOW_HOME="/home/avnish/census_consumer_project/census_consumer_complaint/airflow"To install airflowpip install apache-airflowTo configure databseairflow db initTo create login user for airflowairflow users create  -e avnish@ineuron.ai -f Avnish -l Yadav -p admin -r Admin  -u adminTo start schedulerairflow schedulerTo launch airflow serverairflow webserver -p <port_number>pip install pandas-tfrecordspip install \
  --upgrade --ignore-installed \
  python-snappy==0.5.1 \
  --global-option=build_ext \
  --global-option="-I/usr/local/include" \
  --global-option="-L/usr/local/lib"pip install twine
python setup.py sdist bdist_wheel
twine upload --repository-urlhttps://test.pypi.org/legacy/dist/*
twine upload dist/*To deploy your modelpip install tensorflow-serving-apito inspect modelsaved_model_cli show --dir <dir_path>Above command will return tag setsaved_model_cli show --dir <dir_path> --tag_set <tag_name>Above command will show available model signaturesNext: with tag_set and signature_def info,
we can inspect model input and outputsaved_model_cli show --dir <dir_path> --tag_set <tag_name> --signature_def <SignatureDef Key>To inspect all signature without tag_set and signature_def
saved_model_cli show --dir <dir_path> --allTesting the modelTest model prediction using saved_model_cli with sample input data--input_examples: input data formatted as a tf.Example data structureother param--outdir: by default output will be written in terminal--overwrite: to write into a filetf_debug: run in debug modeTo expose your model as an API using docker image tensorflow/servingdocker pull tensorflow/servingSingle model configurationsudo docker run -p 8500:8500 \
-p 8501:8501\
--volumn <model_dir>:<target_dir>\
-e MODEL_NAME=<model_name>\
-e model_base_path=<target_dir>\
-t tensorflow/serving:latestsudo docker run -p 8500:8500 -p 8501:8501 \
-v  /home/avnish/census_consumer_project/census_consumer_complaint/census_consumer_complaint_data/saved_models:/avnish/my_model \
-e MODEL_NAME=my_model \
-e MODEL_BASE_PATH=/avnish \
-t tensorflow/serving:latestTo inspect docker container directorydocker exec -it <conatiner_name> bash
Package: censusdis
censusdiscensusdis is a package for discovering, loading, analyzing, and computing diversity, integration, and segregation metrics to U.S. Census demographic data.
It is designedto supportevery dataset,every geography, and every year. It's not just about ACS data through the last time the software
was updated and released;to supportall geographies, on and off-spine, not just states, counties, and census tracts;to haveintegrated mapping capabilitiesthat save you time and extra coding;to be intuitive, Pythonic, and fast.Note:For the full README, please visit thecensusdis source repository.Installation and First Examplecensusdis can be installed withpip:pipinstallcensusdisEvery censusdis query needs four things:What data set we want to query.What vintage, or year.What variables.What geographies.Here is an example of how we can use censusdis to download data once we know
those four things.importcensusdis.dataascedfromcensusdis.datasetsimportACS5fromcensusdisimportstatesdf_median_income=ced.download(# Data set: American Community Survey 5-Yeardataset=ACS5,# Vintage: 2022vintage=2022,# Variable: median household incomedownload_variables=['NAME','B19013_001E'],# Geography: All counties in New Jersey.state=states.NJ,county='*')There are many more examples in the tuturial and in the sample notebooks.Tutorial (A Great Place to Start!)For a tutorial, please see thecensusdis-tutorialrepository.
This tutorial was presented atPyData Seattle 2023. If you want to try it out for yourself, the README.md
contains links that let you run the tutorial notebooks live onmybinder.orgin your browser without needing to set up a
local development environment or download or install any code.Tutorial VideoA 86 minutevideoof the tutorial as presented atPyData Seattle 2023is also available.More DetailsFor the full README, lots of maps, demo notebooks, and more, please visit thecensusdis source repository.
Package: census-downloader
Tool for downloading census data for 2020 as a csv in an easy way. Might add other years if there's interest.UsageTo install this script runpip3 install census-downloaderThen to run the script runcensus-downloader --output <OUTPUT_PATH.csv> --columns <COLUMNS YOU WANT GO HERE> --states <STATES TO GO HERE>You can see the spreadsheet atthis linkfor a description of the columns.For example, I rancensus-downloader --output out.csv --columns COUNTY COUSUB SUBMCD ESTATE CONCIT PLACE TRACT BLKGRP BLOCK CBSA MEMI CSA METDIV NECTA NMEMI UA UATYPE UR POP100 INTPTLAT INTPTLON P0010001 P0010002 P0010003 P0010004 P0010005 P0010006 P0010007 P0010008 P0010009 P0030001 P0030002 P0030003 P0030004 P0030005 P0030006 P0030007 P0030008 P0030009Note that I didn't put a--statesargument. By default if you don't provide it it gives you all the statesOn my machine this takes 8min to run, about 8.3GB of RAM and produces a file that is 1.9GB.
Package: census-error-analyzer
census-error-analyzerAnalyze the margin of error in U.S. census dataInstallationpipenvinstallcensus-error-analyzerUsageImport the library.importcensus_error_analyzerTest statistical differenceAre two values, considering their respective margins of error, statistically different? The Census Bureau advises that this test be conducted for all comparisons. This test answers the question and returnsTrueorFalse.Accepts two lists, each expected to be a pair with a value and its margin of error.us_medianage,us_medianage_moe=37.9,0.1nyc_medianage,nyc_medianage_moe=38.4,0.1census_error_analyzer.is_statistically_different((us_medianage,us_medianage_moe),(nyc_medianage,nyc_medianage_moe))TrueThe precise difference can also be accessed. According to the Census Bureau, values greater than 1.0 can be considered to be statistically significant.census_error_analyzer.statistical_difference((us_medianage,us_medianage_moe),(nyc_medianage,nyc_medianage_moe))3.535533905932737Get statistical rangeThe minimum and maximum values in an estimate's statistical range given its margin of error. Expects two arguments: The estimate first. The margin of error second.census_error_analyzer.statistical_range(us_medianage,us_medianage_moe)37.8,38.0Convert to alternative confidence levelsThe margins of error published by the Census are at the 90% confidence level. They can be converted to the 95% and 99% levels using tools in this library.census_error_analyzer.convert_to_95_percent_confidence(3778)4501.446808510638census_error_analyzer.convert_to_99_percent_confidence(3778)5925.373860182372ReferencesThis module was designed to conform with the Census Bureau's April 18, 2018, presentation"Using American Community Survey Estimates and Margin of Error."Prior to publication, the code was reviewed by Brian Dumbacher, a mathematical statistician in the U.S. Census Bureau's Economic Statistical Methods Division.LinksIssues:github.com/datadesk/census-error-analyzer/issuesPackaging:pypi.python.org/pypi/census-error-analyzerTesting:github.com/datadesk/census-error-analyzer/actions
Package: census-extract
The scripts in this python module will upload an Ambry bundle for a release of the Amberican Community Survey to a directory or S3 bucket as CSV files, with one file per summary level per table. The file will also include columns for the non-null values from the geofile.To run these scripts, you must first [install Ambry](http://docs.ambry.io/) and configure it with the ACS bundle remote, and a remote for the CSV files to be written to.However, most users should just use the CSV files that are already written to a public S3 bucket.Using The Public S3 FilesRunning the ScriptsTo run these scripts, you must:# [Install Ambry](http://docs.ambry.io/)
# Add a remote for the census bundles
# Sync the census bundles
# Create a remote to write the CSV files to
Package: census-map-consolidator
census-map-consolidatorCombine Census blocks into new shapes.Installationpipenvinstallcensus-map-consolidatorUsageCreate a list of Census block GEOIDs. That can be simple like this:your_block_list=["060371976001008","060371976001009"]Or read in from a file like this:withopen("test_data/dtla.csv","r")asf:your_block_list=f.read().splitlines()Import our class.fromcensus_map_consolidatorimportBlockConsolidatorPass in the block list as arguments.c=BlockConsolidator(*your_block_list)Consolidate the blocks into a single shape. This can take a minute. Big shapefiles have to be downloaded.c.consolidate()Write out the new shape. You can do GeoJSON.c.write("./your-new-shape.geojson")Or shapefiles.c.write("./your-new-shape.shp")That's it. You can inspect your work however you like. Butgeojsoniois handy.geojsonio<your-new-shape.geojsonLinksIssues:github.com/datadesk/census-map-consolidator/issuesPackaging:pypi.python.org/pypi/census-map-consolidatorTesting:github.com/datadesk/census-map-consolidator/actions
Package: census-map-downloader
census-map-downloaderEasily download U.S. census mapsInstallationpipenvinstallcensus-map-downloaderCommand line usageUsage: censusmapdownloader [OPTIONS] COMMAND [ARGS]...

  Easily download U.S. census maps

Options:
  --data-dir TEXT  The folder where you want to download the data
  --year INTEGER   The vintage of data to download. By default it gets the
                   latest year. Not all data are available for every year.

  --help           Show this message and exit.

Commands:
  blocks                   Download blocks
  congress-carto           Download cartographic congressional districts
  counties                 Download counties
  counties-carto           Download cartographic counties
  countysubdivision        Download cartographic county subdivisions
  legislative-lower-carto  Download cartographic state legislative...
  legislative-upper-carto  Download cartographic state legislative...
  places                   Download places
  states-carto             Download cartographic states
  tracts                   Download tracts
  zctas                    Download ZCTAsExamplesHere's an example of downloading all countiescensusmapdownloadercountiesYou can specify the download directory with --data-dircensusmapdownloader--data-dir./my-special-folder/countiesContributingInstall dependencies for developmentpipenvinstall--devRun testspipenvrunpythontest.pyAdding additional years to a datasetDownloader classes for different geography types are defined in modules of {code}census_map_downloader.geotypes. For example, the downloader for counties is {code}census_map_downloader.geotypes.counties.CountiesDownloader.If the URL and fields in a shapefile are the same as those for years that are already supported, you can just add the year to the {code}YEAR_LISTattribute.If the fields are the same, but the URL changes between groups of years, add logic to the {code}urlproperty method of the downloader classes to alter the URL based on {code}self.year.If the fields and URL change from year to year, consider creating classes for each year and delegating to {code}census_map_downloader.geotypes.tracts.TractsDownloaderis an example of a class that uses this approach.Developing the CLIThe command-line interface is implemented using Click and setuptools. To install it locally for development inside your virtual environment, run the following installation command, asprescribed by the Click documentation.pipenvrunpipinstall--editable.LinksIssues:github.com/datadesk/census-map-downloader/issuesPackaging:pypi.python.org/pypi/census-map-downloaderTesting:github.com/datadesk/census-map-downloader/actions
Package: census-parquet
census-parquetPython tools for creating and maintaining Parquet files fromUS 2020 Census Data.InstallationTo use the data download shell script files first installwget.To install the census-parquet package usepip install census-parquetThis will also install the required Python dependencies which are:clickdaskdask_geopandasgeopandasnumpyopenpyxlpandaspyarrowUsageTo run the census-parquet code simply userun_census_parquetThis runs the following scripts in order:download_boundaries.sh- This script downloads the Census Boundary data needed to runprocess_boundaries.pydownload_population_stats.sh- This script downloads population stat data needed for process_blocks.pydownload_blocks.sh- This script downloads the Census Block data needed to run process_blocks.pyprocess_boundaries.py- This script processes the Census Boundary data and creates parquet files. The parquet files will be output into aboundary_outputsfolder.process_blocks.py- This script processes Census Block data and creates parquet files. The final combined parquet file will have the nametl_2020_FULL_tabblock20.parquet.