Saravanan666/pypi-org huggingface dataset:
Package: udata-ckan
CKAN![Crowdin](https://crowdin.com/project/udata-ckan)CKAN integration for uDataUsageInstall the harvester package in you udata environement:pipinstalludata-ckanThe harvester will be automatically available as a backend choice.DevelopPython dependenciesAssuming you are in an active virtualenv withudatainstalled and in the current project cloned repository directory, install all dependencies using:pipinstall-erequirements/develop.pip##CKAN
==== instanceA docker-compose is availbe to start up a CKAN instance if you want to test your harvester on a custom catalog.TestingTests are located into thetestsfolder and be run with:invtestChangelog3.0.2 (2024-01-25)Usedatetime.utcnowto make sure to handle utc datetimes#246Removevalid_aton deprecated GeoZones#2483.0.1 (2023-03-17)Use id as remote_id ASAP for better error handling#239Update mongoDB in CI#242Refactor tests to mock CKAN results instead of starting a CKAN instance#245Upgrade CKAN images used in docker-compose#244Fix resource.published that is not needed anymore#2433.0.0 (2022-11-14)warning:Breaking changeUse harvest dynamic field introduced in udata 5#2272.0.1 (2022-09-01)Replace mongo legacy image in CI#219Make revision_id optional to match latest CKAN versions#2202.0.0 (2020-03-11)Migrate to python3 🐍#1101.3.0 (2020-01-06)DKAN support#1291.2.3 (2019-05-29)Always fill extras.remote_url#1031.2.2 (2019-05-24)Max out package_search rows limit#1001.2.1 (2019-05-24)Fix filetype (always remote)#981.2.0 (2018-10-02)Support both inclusion and exclusion filters#42Localization support#43Test the minimum accepted CKAN dataset payload and make theextrasproperty optional#57Improved error handling (support details in JSON responses, also handle raw quoted strings and HTML)#561.1.1 (2018-06-15)Only storeurlfield inremote_urlextra if this is an URL otherwise store it inckan:source#30Properly handle geometry errors#31Improve extras parsing#32:
- Skip empty extras
- Parse update frequencies as RDF URI or udata frequency identifier
- Parsespatial-textmatching a known zone name or slug
- Store unknownspatial-uri,spatial-textandfrequencyasckan:spatial-uri,ckan:spatial-textandckan:frequency1.1.0 (2018-06-06)Test against a real CKAN instance#23Allows to filter on Organizations and Tags#26Registerckan:prefixed extras#281.0.1 (2018-03-13)Fix packaging#2Make use ofudata pytest plugin#3Enable theckanplugin in test (plugin needs to be enabled to use the harvester)#81.0.0 (2017-10-20)Initial release
Package: udata-croquemort
uData-croquemortThis plugin provides integration between uData and Croquemort link checker.Compatibilityudata-croquemortrequires Python 2.7+ anduData.InstallationInstalluData.Remain in the same virtual environment (for Python) and use the same version of npm (for JS).Installudata-croquemort:pipinstalludata-croquemortModify your local configuration file ofudata(typically,udata.cfg) as following:PLUGINS=['croquemort']LINKCHECKING_DEFAULT_LINKCHECKER='croquemort'CROQUEMORT_URL='http://localhost:8000'CROQUEMORT_NB_RETRY=10CROQUEMORT_DELAY=1Changelog2.0.1 (2020-06-17)Expose new HTTP headers related to content (content-type,content-length,charset,content-md5,content-disposition)#1332.0.0 (2020-03-11)Migrate to python3 🐍#931.0.4 (2019-05-20)Do not check FTP(S) urls#861.0.3 (2018-07-30)Handle croquemort response w/o updated attr#311.0.2 (2018-03-13)Switch to pytest1.0.1 (2017-10-20)Fix setup.py version1.0.0 (2017-10-20)Stable version0.0.1 (2017-10-03)Initial release
Package: udata-data-fair
# DataFair plugin for uDataUse DataFair embeds to preview data## UsageInstall the plugin package in you udata environement:`bash pip installudata-data-fair`Then activate it in yourudata.cfg:`python PLUGINS =['udata-data-fair']`## Publish` python setup.py sdist bdist_wheel twine uploaddist/*.tar.gz`# Changelog## Current (in progress)Initial release
Package: udata-geoplatform
uData Geoplatform connectorConnects uData with Geoplatform (geo.data.gouv.fr)UsageInstall the plugin package in you udata environement:pipinstalludata-geoplatformThen activate it in yourudata.cfg:PLUGINS=['geoplatform']ConfigurationYou can control this plugin’s behavior with the followingudata.cfgparameters:GEOPLATFORM_URL: The URL to yourgeoplatforminstance (without trailing slash).ex:https://geo.data.gouv.frChangelog2.0.0 (2020-03-11)Migrate to python3 🐍#611.0.1 (2018-08-03)Pass current locale to preview URL#31.0.0 (2018-06-06)Initial release
Package: udata-hydra-csvapi
Api-tabularThis connects toudata-hydraand serves the converted CSVs as an API.Run locallyStartudata-hydraviadocker compose.Launch this project:dockercomposeupYou can now access the raw postgrest API onhttp://localhost:8080.Now you can launch the proxy (ie the app):poetry install
poetry run adev runserver -p8005 api_tabular/app.py        # Api related to apified CSV files by udata-hydra
poetry run adev runserver -p8005 api_tabular/metrics.py    # Api related to udata's metricsAnd query postgrest via the proxy using aresource_id, cf below. Test resource_id is27d469ff-9908-4b7e-a2e0-9439bb38a395APIMeta informations on resourcecurlhttp://localhost:8005/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/{"created_at":"2023-02-11T11:44:03.875615+00:00","url":"https://data.toulouse-metropole.fr//explore/dataset/boulodromes/download?format=csv&timezone=Europe/Berlin&use_labels_for_header=false","links":[{"href":"/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/profile/","type":"GET","rel":"profile"},{"href":"/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/data/","type":"GET","rel":"data"}]}Profile (csv-detective output) for a resourcecurlhttp://localhost:8005/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/profile/{"profile":{"header":["geo_point_2d","geo_shape","ins_nom","ins_complexe_nom_cplmt","ins_codepostal","secteur","..."]},"...":"..."}Data for a resource (ie resource API)curlhttp://localhost:8005/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/data/{"data":[{"__id":1,"geo_point_2d":"43.58061543292057,1.401751073689455","geo_shape":{"coordinates":[[1.401751073689455,43.58061543292057]],"type":"MultiPoint"},"ins_nom":"BOULODROME LOU BOSC","ins_complexe_nom_cplmt":"COMPLEXE SPORTIF DU MIRAIL","ins_codepostal":31100,"secteur":"Toulouse Ouest","quartier":6.3,"acces_libre":null,"ins_nb_equ":1,"ins_detail_equ":"","ins_complexe_nom":"","ins_adresse":"","ins_commune":"","acces_public_horaires":null,"acces_club_scol":null,"ins_nom_cplmt":"","ins_id_install":""}],"links":{"next":"/api/resources/60963939-6ada-46bc-9a29-b288b16d969b/data/?page=2&page_size=1","prev":null,"profile":"/api/resources/60963939-6ada-46bc-9a29-b288b16d969b/profile/"},"meta":{"page":1,"page_size":1,"total":2}}This endpoint can be queried with the following operators as query string:# sort by column
column_name__sort=asc
column_name__sort=desc

# contains
column_name__contains=word

# exacts
column_name__exact=word

# less
column_name__less=12

# greater
column_name__greater=12Pagination is made throught queries withpageand page_size:curl http://localhost:8005/api/resources/27d469ff-9908-4b7e-a2e0-9439bb38a395/data/?page=2&page_size=30
Package: udata-koumoul
udata-koumoulTheme for Koumoul's demo opendata portalUsageInstall the theme package in you udata environement:.. code-block:: bashpip install udata-koumoulThen, define the installed theme as current in youudata.cfg:.. code-block:: pythonTHEME = 'koumoul'DevelopmentThere is adocker-composeconfiguration to get started fast.
Just run:.. code-block:: bashdocker-compose upThen go tohttp://localhost:7000to connect to the development server
with live reload.Publish.. code-block:: Nonepython setup.py sdist bdist_wheel
twine upload dist/*ChangelogCurrentInitial release
Package: udata-metrics
udata-metricsThis plugin adds views to display metrics on datasets, reuses, organizations and site dashboard pages.
It uses hook logic based on udata-front hooks.
We feed these views with data from:
* an optional metrics API for traffic (views, download, external links)
* mongo for stock values (with an aggregation pipeline)
A job can be scheduled to inject traffic metrics in udata's objects metrics.InstallationInstalludata.Remain in the same virtual environment (for Python).Installudata-metrics:pipinstalludata-metricsModify your local configuration file ofudata(typically,udata.cfg) as following:PLUGINS=['metrics']METRICS_API='http://localhost:8005/api'Changelog2.0.1 (2023-12-01)Improve metrics layout#11Specify that statistics are for the yearMove datasets graph on the second row in dashboard statsUse current month value (and not variation)Prevent upper or lowercasing by removing any text transformation2.0.0 (2023-11-21):warning: New metrics logic#8Add metrics views (using template hook)Use optional metrics API (with job to update metrics on objects)Remove Influx logicReplace mongo legacy image in CI#61.0.2 (2020-07-01)Add time condition in the queries to retrieve only last 24 hours results#41.0.1 (2020-05-13)Nothing yet1.0.0 (2020-05-12)Initial release
Package: udata-ods
OpenDataSoft udata integration![Crowdin](https://crowdin.com/project/udata-ods)OpenDataSoft integration for uDataUsageInstall the harvester package in you udata environement:pipinstalludata-odsChangelog4.0.0 (2024-01-09)warning:Add migration from ODS to DCAT#247- All ODS harvest sources are converted to DCAT
- Datasets are updated with new remote_id and new resources urlsAdd migration that removes duplicate resources#2483.0.1 (2023-02-06)Fix can_explore on dataset with both ODS and non-ODS resources#2443.0.0 (2022-11-14)warning:Breaking changeUse harvest dynamic field introduced in udata 5#234Store geojson format instead of json when relevant#212Replace mongo legacy image in CI#2212.1.0 (2020-10-16)Link exports (resources) to a version of the file with machine names as headers, instead of labels#1692.0.1 (2020-03-24)Usemetas`modified]->dataset.last_modified[#148 <https://github.com/opendatateam/udata-ods/pull/148>`_2.0.0 (2020-03-11)Migrate to python3 🐍#911.2.4 (2019-05-29)Fill extras.remote_url#1111.2.3 (2019-03-27)Fetch each dataset metadata on processing instead of once on initialization (ensures fresher metadata)#981.2.2 (2019-02-01)Ensure dataset with only attachments or alternative exports are properly harvested#861.2.1 (2018-10-02)Fix dependency on udata 1.6.0 (instead of 1.6.0.dev)1.2.0 (2018-10-02)Initial filtering support#52#61Optionnal Inspire dataset harvesting#61Fix missing translations#531.1.0 (2018-06-06)Initial i18n support#20Do not export SHP when above records limit#26Initial resources preview support#29#34Useudata.frontend.markdown:parse_html()to parse description#35Attachments support#36Registerods:prefixed extras#421.0.1 (2018-03-13)Fix URLs handling (no double slashes)#2Fix packaging#3Make use ofudata pytest plugin#4Alternative exports support#141.0.0 (2017-10-20)Initial release
Package: udata-recommendations
udata-recommendationsThis plugin acts as a bridge between uData and a recommendation system.In our case (data.gouv.fr), it's a set of scripts living herehttps://github.com/etalab/piwik-covisits.Recommendations are stored on datasets. Recommendations can come from various sources and are stored in a descending order, according to the provided score (from 1 to 100). The top recommendations are displayed at the bottom on the dataset page.Compatibilityudata-recommendationsrequires Python 3.7+ anduData.InstallationInstalluData.Remain in the same virtual environment (for Python).Installudata-recommendations:pipinstalludata-recommendationsModify your local configuration file ofudata(typically,udata.cfg) as following:PLUGINS=['recommendations']RECOMMENDATIONS_SOURCES={'source-name':'https://path/to/recommendations.json','other-source':'https://path/to/other/recommendations.json',}RECOMMENDATIONS_NB_RECOMMENDATIONS=4RECOMMENDATIONS_SOURCES: A key-value dictionary of recommendation sources and URLs to fetch.Default:{}RECOMMENDATIONS_NB_RECOMMENDATIONS: The maximum number of recommendations to display on the dataset page.Default:4UsageAdding recommendationsYou can fetch and store recommendations as a task, using your configuration inRECOMMENDATIONS_SOURCES, on a schedule if needed. By default, previous recommendations are cleaned before the importing new ones, but you're in control.udatajobrunrecommendations-add# Don't clean each source before importing new recommendationsudatajobrunrecommendations-addshould_clean=falseDeleting recommendationsTo clean all recommendations, you can run the following task.udatajobrunrecommendations-cleanExpectationsThis plugin expects the following format to provide datasets recommendations:[{"id":"dataset-id","recommendations":[{"id":"dataset-slug-1","score":100},{"id":"5ef1fe80f50446b8f41ba691","score":1}]},{"id":"dataset-id2","recommendations":[{"id":"5ef1fe80f50446b8f41ba691","score":50}]}]Dataset IDs can be IDs or slugs. Scores should be between1and100, inclusive. You can validate your JSON using aJSON Schema.Changelog3.1.5 (2023-11-21)Update Matomo content tracking data-attributes#263Upgrade test and develop dependencies#2643.1.4 (2023-03-07)Update and compile translations#261#2623.1.3 (2023-03-02)Recommendations for new dataset page#2563.1.2 (2022-12-15)Update dataset and reuse recommendations to match new udata-front layout2073.1.1 (2022-09-01)Replace mongo legacy image in CI#226Store unique recommendations in extras#2393.1.0 (2021-09-16)Change udata-gouvfr dependency to udata-front following renaming#1883.0.0 (2021-08-12)Ensure compatibility with udata3 by changing imports and style#1832.2.0 (2020-11-30)Add reuses support#1532.1.1 (2020-10-16)Ignore recommendation of dataset itself#1472.1.0 (2020-08-25)Add score to recommendations and support multiple recommendation sources#1422.0.0 (2020-03-11)udata 2.0 / Python 3 support#95Support new hooks format#961.0.1 (2018-08-03)Nothing yet1.0.0 (2018-06-06)Allow slug instead of id for datasets#8Initial release
Package: udata-schema-gouvfr
# udata-schema-gouvfrThis udata plugin provides an integration with [schema.data.gouv.fr](https://schema.data.gouv.fr)## FeaturesAdds a schema button to resources with a schemaDisplays a modal when clicking this button providing links to documentation and validation## UsageInstall the plugin package in your udata environment:`bash pip installudata-schema-gouvfr`Then activate it in yourudata.cfg:`python PLUGINS =['schema-gouvfr']`## ConfigurationYou can control this plugin behavior with the followingudata.cfgparameters:`SCHEMA_GOUVFR_VALIDATA_URL`: the URL to your [Validata](https://validata.fr/) instance (without trailing slash).ex:https://validata.etalab.studio`SCHEMA_GOUVFR_IRVE_STABLE_RESOURCE_URL`: the permalink to the consolidated [IRVE dataset](irve-dataset)’s latest resource.ex:https://www.data.gouv.fr/fr/datasets/r/50625621-18bd-43cb-8fde-6b8c24bdabb3## JobsThis plugin declares jobs.`udata job run set-irve-schemas`: sets theschemaattribute of resources that are consolidated as part of [the national IRVE dataset](irve-dataset)[irve-dataset]:https://www.data.gouv.fr/fr/datasets/fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques/# Changelog## 1.3.2 (2021-05-28)Fix links in modal (Validata and data.gouv facet)Fix to function which detect if resource has schema or not## 1.3.1 (2021-05-25)Add venv’s activation in circle’s configuration file for publish step [#8](https://github.com/etalab/udata-schema-gouvfr/pull/8)## 1.3.0 (2021-05-25)Add version schema if exists in modal [#7](https://github.com/etalab/udata-schema-gouvfr/pull/7)## 1.2.0 (2020-10-16)Link to datasets with the same schema from the schema modal [#4](https://github.com/etalab/udata-schema-gouvfr/pull/4)## 1.1.0 (2020-08-25)Add a job to set theschemaattribute to the IRVE schema on multiple resources [#3](https://github.com/etalab/udata-schema-gouvfr/pull/3)## 1.0.0 (2020-08-05)Initial release
Package: udata-search-service
udata-search-serviceA search service for udata.
The idea is to have search service separated from the udata MongoDB.
The indexation update is made using real-time HTTP messages.See the following architecture schema:Getting startedYou can follow this recommended architecture for your code:$WORKSPACE
├── fs
├── udata
│   ├── ...
│   └── setup.py
│		└── udata.cfg
├── udata-front
│   ├── ...
│   └── setup.py
└── udata-search-service
    ├── ...
    └── pyproject.tomlClone the repository:cd $WORKSPACE
git clone git@github.com:opendatateam/udata-search-service.gitStart the different services using docker-compose:cd udata-search-service
docker-compose upThis will start:an elasticsearcha search appInitialize the elasticsearch indices on setup.# Locally
udata-search-service init-es

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service init-es'This will create the following indices:{UDATA_INSTANCE_NAME}-dataset-{yyyy}-{mm}-{dd}-{HH}-{MM}{UDATA_INSTANCE_NAME}-reuse-{yyyy}-{mm}-{dd}-{HH}-{MM}{UDATA_INSTANCE_NAME}-organization-{yyyy}-{mm}-{dd}-{HH}-{MM}Configure your udata to use the search service, by updating the following variables in your udata.cfg.
Ex in local:SEARCH_SERVICE_API_URL = 'http://127.0.0.1:5000/api/1/'Usingudata, when you modify objects,
indexation messages will be sent to the search app and will be consumed by the API.
If you want to reindex your local mongo base in udata, you can run:cd $WORKSPACE/udata/
source ./venv/bin/activate
udata search indexMake sure to have the corresponding UDATA_INSTANCE_NAME specified in your udata settings.After a reindexation, you'll need to change the alias by using the following command:# Locally
udata-search-service set-alias <index-suffix>

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service set-alias <index-suffix>'You can query the search service with the search service api, ex:http://localhost:5000/api/1/datasets/?q=toilettes%20à%20rennesDevelopmentYou can create a virtualenv, activate it and install the requirements with the following commands.python3 -m venv venv
source venv/bin/activate
make deps
make installYou can start the web search service with the following command:udata-search-service runDeploymentThe project depends onElasticSearch7.16.Elasticsearch requires theAnalysis ICUplugin for your specific version.
On Debian, you can take a look at theseinstructionsfor installation.TroubleshootingIf the elasticsearch service exits with an error 137, it is killed due to out of memory error. You should read the following points.If you are short on RAM, you can limit heap memory by settingES_JAVA_OPTS=-Xms750m -Xmx750mas environment variable when starting the elasticsearch service.If you are on MAC and still encounter RAM memory issues, you should increase Docker limit memory to 4GB instead of default 2GB.If you are on Linux, you may need to double the vm.max_map_count. You can set it with the following command:sysctl -w vm.max_map_count=262144.If you are on Linux, you may encounter permissions issues. You can either create the volume or change the user to the current user usingchown.
Package: udatastore
μDatastore: ODM for Google Cloud DatastoreImplemented as framework in theμMongolibrary (a python ODM for MongoDB)While working Google Cloud Datastore, most of our code uses thegoogle-cloud-datastorepackage,
for creating, updating and querying entities. We often find ourselves writing code to export objects
todatastore.Entity, and import from entities to objects. On AppEngine, there isndbwhich avoids this,
so the idea emerged to create an ODM for datastore ourselves.Having worked with μMongo before, a prototype was created which implements an additional framework based on
following conventions:A datastore partition (project/namespace) corresponds to a mongo databaseA datastore entity kind corresponds to a mongo collectionThe datastore__key__field corresponds to the mongo_idfieldInstallpip install udatastoreExamplefromdatetimeimportdatetimefromgoogle.cloudimportdatastorefromudatastoreimportDataStoreInstancefromumongoimportDocument,fields,validatedb=datastore.Client(project="dummy",namespace='abcd')instance=DataStoreInstance()instance.init(db)@instance.registerclassUser(Document):email=fields.EmailField(required=True,unique=True)birthday=fields.DateTimeField(validate=validate.Range(min=datetime(1900,1,1)))friends=fields.ListField(fields.ReferenceField("User"))goku=User(email='goku@sayen.com',birthday=datetime(1984,11,20))goku.commit()vegeta=User(email='vegeta@over9000.com',friends=[goku])vegeta.commit()vegeta.friends# <object umongo.data_objects.List([<object udatastore.reference.DataStoreReference(document=User, pk=<Key('User', 4476), project=dummy>)>])>vegeta.dump()# {'email': 'vegeta@over9000.com', 'id': '4477', 'friends': [<Key('User', 4476), project=dummy>]}User.find_one({"email":'goku@sayen.com'})# <object Document __main__.User({'email': 'goku@sayen.com', 'id': 4474,#                                 'friends': <object umongo.data_objects.List([])>,#                                 'birthday': datetime.datetime(1984, 11, 20, 0, 0)})>Limitations:Not all features of μMongo are available or work exactly the same in μDatastore:No indexesWe do not currently support all field typesDatastore converts anydatetimeto UTC timezone. For now, μDatastore overrides this behaviour and always works with
unaware datetimes.We bring our ownReferenceFieldimplementation. You are free to useumongo.fields.ReferenceField, as theDataStoreBuilderwill automatically replace these fields with our implementation. We also disable the io_validation on reference fields.
Because datastore is eventually consistent this may report errors when creating references to previously created entities.
Package: udata-transport
udata-transportThis plugin acts as a bridge betweenudataandtransport.data.gouv.frsystem.Compatibilityudata-transportrequires Python 3.7+ andudata.InstallationInstalludata.Remain in the same virtual environment (for Python).Installudata-transport:pipinstalludata-transportModify your local configuration file ofudata(typically,udata.cfg) as following:PLUGINS=['transport']TRANSPORT_DATASETS_URL='https://path/to/transport.data.gouv.fr/datasets/api'TRANSPORT_DATASETS_URL: The URL of the API endpoint listing datasets on transport.data.gouv.fr.Default:https://transport.data.gouv.fr/api/datasetsUsageMapping datasetThe mapping is done by a job, runnable by hand or scheduled.
The job is idempotent, previous URLs are cleaned before mapping new ones.udatajobrunmap-transport-datasetsThis plugin will store the mapping URL in the dataset extrastransport:urlkey:{"id":"dataset-id","extras":{"transport:url":"https://path/to/datasets"}}Changelog1.0.1 (2024-02-15)Replace mongo legacy image in CI#2Clear datasets only after successful request#31.0.0 (2022-03-31)Initial version of udata-transport#1
Package: udata-youckan
uData-youckanThis plugin provide integration betweenuDataandYouCKANCompatibilityudata-youckanrequires Python 2.7+ anduData.InstallationInstalluData.Remain in the same virtual environment (for Python) and use the same version of npm (for JS).Installudata-youckan:pipinstalludata-youckanConfigurationIn order to use YouCKAN as authentication provider, you need to enable the plugin
and add the following mandatory parameters to you uData configuration
(typically,udata.cfg) as following:PLUGINS=['youckan']YOUCKAN_URL='https://your.youckan.url/'YOUCKAN_CONSUMER_KEY='your-youckan-client-key',YOUCKAN_CONSUMER_SECRET='your-youckan-secret-key'Changelog1.0.0 (2017-02-16)Use email to fetch user instead of slug (#7)Keep active state synchronized (#6)0.9.1 (2017-01-10)Fix packaging0.9.0 (2017-01-10)First published release
Package: udbnn
Experiment to determine whetever a large batch-size can be helpful with extremely umbalanced datasets.How do I install this package?As usual, just download it using pip:pipinstalludbnnTests CoverageSince some software handling coverages sometime get slightly different results, here’s three of them:How do I run the experiments?Since the experiments take quite a bit to run, I suggest you to run them while in a TMUX-like environment. If available, you should consider using a computer with a tensorflow-compatible GPU.Then just run with a python shell:fromudbnnimportrunrun("dataset")