Package: cdp-data
cdp-dataData Utilities and Processing Generalized for All CDP InstancesInstallationStable Release:pip install cdp-dataDevelopment Head:pip install git+https://github.com/CouncilDataProject/cdp-data.gitDocumentationFor full package documentation please visitcouncildataproject.github.io/cdp-data.QuickstartPulling DatasetsInstall basics:pip install cdp-dataTranscripts and Session Datafromcdp_dataimportCDPInstances,datasetsds=datasets.get_session_dataset(infrastructure_slug=CDPInstances.Seattle,start_datetime="2021-01-01",store_transcript=True,)Transcript Schema and UsageIt may be useful to look at ourtranscript model documentation.Transcripts can be read into memory and processed as an object:fromcdp_backend.pipeline.transcript_modelimportTranscript# Read the file as a Transcript objectwithopen("transcript.json","r")asopen_f:transcript=Transcript.from_json(open_f.read())# Navigate the objectforsentenceintranscript.sentences:if"clerk"insentence.text.lower():print(f"{sentence.index},{sentence.start_time}: '{sentence.text}')If you do not want to do this processing in Python or prefer to work with
a DataFrame, you can convert transcripts to DataFrames like so:fromcdp_dataimportdatasets# assume that transcript is the same transcript as the prior code snippetsentences=datasets.convert_transcript_to_dataframe(transcript)You can also do this conversion (and storage of the coverted transcript) for
all transcripts in a session dataset during dataset construction with thestore_transcript_as_csvparameter.fromcdp_dataimportCDPInstances,datasetsds=datasets.get_session_dataset(infrastructure_slug=CDPInstances.Seattle,start_datetime="2021-01-01",store_transcript=True,store_transcript_as_csv=True,)This will store the transcript for each session as both JSON and CSV.Voting Datafromcdp_dataimportCDPInstances,datasetsds=dataset.get_vote_dataset(infrastructure_slug=CDPInstances.Seattle,start_datetime="2021-01-01",)Data Definitions and SchemaPlease refer to ourdatabase schemaand ourdatabase model definitionsfor more information on CDP generated and archived data is structured.Saving DatasetsBecause we heavily rely on our database models for database interaction,
in many cases, we default to returning the fullfireo.models.Modelobject
as column values.These objects cannot be immediately stored to disk so we provide a helper to
replace all model objects with their database IDs for storage.This can be done directly if you already have a dataset you have been working with:fromcdp_dataimportdatasets# data should be a pandas dataframedataset.save_dataset(data,"data.csv")Or this can be premptively be done during dataset construction:fromcdp_dataimportCDPInstances,dataset# both get_session_dataset and get_vote_dataset# have a `replace_py_objects` parametersessions=datasets.get_session_dataset(infrastructure_slug=CDPInstances.Seattle,replace_py_objects=True,)votes=datasets.get_vote_dataset(infrastructure_slug=CDPInstances.Seattle,replace_py_objects=True,)Plotting and AnalysisInstall plotting support:pip install cdp-data[plot]Ngram Usage over Timefromcdp_dataimportCDPInstances,keywords,plotsngram_usage=keywords.compute_ngram_usage_history(CDPInstances.Seattle,start_datetime="2022-03-01",end_datetime="2022-10-01",)grid=plots.plot_ngram_usage_histories(["police","housing","transportation"],ngram_usage,lmplot_kws=dict(# extra plotting paramscol="ngram",hue="ngram",scatter_kws={"alpha":0.2},aspect=1.6,),)grid.savefig("seattle-keywords-over-time.png")DevelopmentSeeCONTRIBUTING.mdfor information related to developing the code.MIT license