Package: openrefine-client
OpenRefine Python Client with extended command line interfaceTheOpenRefine Python Client from PaulMakepeaceprovides a library for communicating with anOpenRefineserver.
This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
It is also available via Docker Hub, PyPI and Binder.works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1DownloadOne-file-executables:Windows:openrefine-client_0-3-10_windows.exe(~5 MB)macOS:openrefine-client_0-3-10_macos(~5 MB)Linux:openrefine-client_0-3-10_linux(~5 MB)ForDockercontainers, nativePythoninstallation and freeBinderon-demand server see the corresponding chapters below.PeekA short video loop that demonstrates the basic features (list, create, apply, export):UsageEnsure you haveOpenRefinerunning (i.e. available athttp://localhost:3333oranother URL).To use the client:Open a terminal pointing to the folder where you havedownloadedthe one-file-executable (e.g. Downloads in your home directory).Windows: Open PowerShell and enter following commandcd~\DownloadsmacOS: Open Terminal (Finder > Applications > Utilities > Terminal) and enter following commandcd~/DownloadsLinux: Open terminal app (Terminal, Konsole, xterm, ...) and enter following commandcd~/DownloadsMake the file executable.Windows: not necessarymacOS:chmod+xopenrefine-client_0-3-10_macosLinux:chmod+xopenrefine-client_0-3-10_linuxExecute the file.Windows:.\openrefine-client_0-3-10_windows.exemacOS:./openrefine-client_0-3-10_macosLinux:./openrefine-client_0-3-10_linuxUsing tab completion and command history is highly recommended:autocomplete filenames: enter a few characters and press↹recall previous command: press↑Basic commandsExecute the client by entering its filename followed by the desired command.The following example will download two small files (duplicates.csvandduplicates-deletion.json) into the current directory and will create a new OpenRefine project from file duplicates.csv.Download example data (--download) and create project from file (--create):Windows:.\openrefine-client_0-3-10_windows.exe--download"https://git.io/fj5hF"--output=duplicates.csv
.\openrefine-client_0-3-10_windows.exe--download"https://git.io/fj5ju"--output=duplicates-deletion.json
.\openrefine-client_0-3-10_windows.exe--createduplicates.csvmacOS:./openrefine-client_0-3-10_macos--download"https://git.io/fj5hF"--output=duplicates.csv
./openrefine-client_0-3-10_macos--download"https://git.io/fj5ju"--output=duplicates-deletion.json
./openrefine-client_0-3-10_macos--createduplicates.csvLinux:./openrefine-client_0-3-10_linux--download"https://git.io/fj5hF"--output=duplicates.csv
./openrefine-client_0-3-10_linux--download"https://git.io/fj5ju"--output=duplicates-deletion.json
./openrefine-client_0-3-10_linux--createduplicates.csvOther commands:list all projects:--listshow project metadata:--info "duplicates"export project to terminal:--export "duplicates"applyrules from json file:--apply duplicates-deletion.json "duplicates"export project to file:--export --output=deduped.xls "duplicates"delete project:--delete "duplicates"Getting helpCheck--helpfor further options.Please file anissueif you miss some features in the command line interface or if you have tracked a bug.
And you are welcome to ask any questions!Change URLBy default the client connects to the usual URL of OpenRefinehttp://localhost:3333.
If your OpenRefine server is running somewhere else then you may set hostname and port with additional command line options (e.g.http://example.com):set host:-H example.comset port:-P 80TemplatingThe OpenRefineTemplatingsupports exporting data in any text format (i.e. to construct JSON or XML).
The graphical user interface offers four input fields:prefixrow templatesupportsGRELinside two curly brackets, e.g.{{jsonize(cells["name"].value)}}row separatorsuffixThis templating functionality is available via the openrefine-client command line interface.
It even provides an additional feature for splitting results into multiple files.To try out the functionality create another project from the example file above.--createduplicates.csv--projectName=advancedThe following example code will export...the columns "name" and "purchase" in JSON formatfrom the project "advanced"for rows matching the regex text filter^F$in column "gender"macOS/Linux Terminal (multi-line input with\):"advanced"\--prefix='{ "events" : ['\--template='    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }'\--rowSeparator=','\--suffix='] }'\--filterQuery='^F$'\--filterColumn='gender'Windows PowerShell (multi-line input with`; quotes needs to be doubled):"advanced"`--prefix='{ ""events"" : ['`--template='    { ""name"" : {{jsonize(cells[""name""].value)}}, ""purchase"" : {{jsonize(cells[""purchase""].value)}} }'`--rowSeparator=','`--suffix='] }'`--filterQuery='^F$'`--filterColumn='gender'Add the following options to the last command (recall with↑) to store the results in multiple files.
Each file will contain the prefix, an processed row, and the suffix.--output=advanced.json--splitToFiles=trueFilenames are suffixed with the row number by default (e.g.advanced_1.json,advanced_2.jsonetc.).
There is another option to use the value in the first column instead:--output=advanced.json--splitToFiles=true--suffixById=trueBecause our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g.advanced_melanie.white@example2.edu.json).
When using this option, the first column should contain unique identifiers.Append data to an existing projectOpenRefine does not support appending rows to an existing project.
As long as thefeature requestis not yet implemented, you can use the openrefine-client to script a workaround:export existing project as csvput old and new data into a zip archivecreate new project by importing the zip archiveHere is an example that replaces the existing project:openrefine-client --export myproject --output old.csv
openrefine-client --delete myproject
zip combined.zip old.csv new.csv
openrefine-client --create combined.zip --format csv --projectName myprojectNote that the project id will change.
If you want to distinguish between old and new data, you can use the additional flag includeFileSources:openrefine-client --create combined.zip --format csv --projectName myproject --includeFileSources trueSee alsoLinux Bash script to run OpenRefine in batch mode (import, transform, export):openrefine-batchJupyter notebook demonstrating usage in Linux BashUse caseHOS-MetadataTransformations: Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.Use caseData processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.Dockerfelixlohmeier/openrefine-clientdockerpullfelixlohmeier/openrefine-client:v0.3.10Option 1: Dockerized clientRun client and mount current directory as workspace:dockerrun--rm--network=host-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10The docker option--network=hostallows you to connect to a local or remote OpenRefine via the host network:list projects on default URL (http://localhost:3333)dockerrun--rm--network=host-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10--listlist projects on a remote server (http://example.com)dockerrun--rm--network=host-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hexample.com-P80--listUsage: same commands as explained above (seeBasic CommandsandAdvanced Templating)Option 2: Dockerized client and dockerized OpenRefineRun openrefine-client linked to a dockerized OpenRefine (felixlohmeier/openrefine):Create docker networkdockernetworkcreateopenrefineRun server (will be available athttp://localhost:3333)dockerrun-d-p3333:3333--network=openrefine--name=openrefine-serverfelixlohmeier/openrefine:3.4.1Run client with somebasic commands: 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete projectdockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10--download"https://git.io/fj5hF"--output=duplicates.csv
dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10--download"https://git.io/fj5ju"--output=duplicates-deletion.json
dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--createduplicates.csv
dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--list
dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--info"duplicates"dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--export"duplicates"dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--applyduplicates-deletion.json"duplicates"dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--export"duplicates"dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--export--output=deduped.xls"duplicates"dockerrun--rm--network=openrefine-v${PWD}:/data:zfelixlohmeier/openrefine-client:v0.3.10-Hopenrefine-server--delete"duplicates"Stop and delete server:dockerstopopenrefine-server
dockerrmopenrefine-serverDelete docker network:dockernetworkrmopenrefineCustomize OpenRefine server:If you want to add an OpenRefine startup option you need to repeat the default commands (cf.Dockerfile)-i 0.0.0.0sets OpenRefine to be accessible from outside the container, i.e. from host-d /datasets OpenRefine workspaceExample forallocating more memoryto OpenRefine with additional option-m 4Gdockerrun-d-p3333:3333--network=openrefine--name=openrefine-serverfelixlohmeier/openrefine:3.4.1-i0.0.0.0-d/data-m4GThe OpenRefine version is defined by the docker tag.
Check theDockerHub repositoryfor available tags.
Example for OpenRefine2.8with same options as above:dockerrun-d-p3333:3333--network=openrefine--name=openrefine-serverfelixlohmeier/openrefine:2.8-i0.0.0.0-d/data-m4GIf you want OpenRefine to read and write persistent data in host directory (i.e. store projects) you can mount the container path/data. Example for host directory/home/felix/refine:dockerrun-d-p3333:3333-v/home/felix/refine:/data:z--network=openrefinename=openrefine-serverfelixlohmeier/openrefine:2.8-i0.0.0.0-d/data-m4GSee also:GitHub Repositoryfor docker containerfelixlohmeier/openrefineLinux Bash script to run OpenRefine in batch mode (import, transform, export) with docker containers:openrefine-batch-docker.shPythonopenrefine-client(requires Python 2.x)python2-mpipinstallopenrefine-client--userThis will install the packageopenrefine-clientcontaining modules ingoogle.refine.A command line scriptopenrefine-clientwill also be installed.Option 1: command line scriptopenrefine-client--helpUsage: same commands as explained above (seeBasic CommandsandAdvanced Templating)Option 2: using cli functions in Python 2.x environmentImport module cli:fromgoogle.refineimportcliChange URL (if necessary):cli.refine.REFINE_HOST='localhost'cli.refine.REFINE_PORT='3333'Help screen:help(cli)Commands:download (e.g. example data):cli.download('https://git.io/fj5hF','duplicates.csv')cli.download('https://git.io/fj5ju','duplicates-deletion.json')list projects:cli.ls()create project:p1=cli.create('duplicates.csv')show metadata:cli.info(p1.project_id)apply rules from file to project:cli.apply(p1.project_id,'duplicates-deletion.json')export project to terminal:cli.export(p1.project_id)export project to file in xls format:cli.export(p1.project_id,'deduped.xls')export templating (seeAdvanced Templatingabove):cli.templating(p1.project_id,prefix='''{ "events" : [''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',rowSeparator=''',''',suffix='''] }''')delete project:cli.delete(p1.project_id)Option 3: the upstream wayThis fork can be used in the same way as the upstreamPython client library.Some functions in the python client library are not yet compatible with OpenRefine >=3.0 (cf.issue #19 in refine-client-py).Import module refine:fromgoogle.refineimportrefineServer Commands:set up connection:server1=refine.Refine('http://localhost:3333')show version:server1.server.get_version()server1.server.versionlist projects:server1.list_projects()pretty print the returned dict with json.dumps:importjsonprint(json.dumps(server1.list_projects(),indent=1))create project:server1.new_project(project_file='duplicates.csv')create and open the returned project in one step:project1=server1.new_project(project_file='duplicates.csv')Project commands:open project:project1=server1.open_project('1234567890123')print full URL to project:project1.project_url()list columns:project1.columnscompute text facet on first column (fails with OpenRefine >=3.2):project1.compute_facets(facet.TextFacet(project1.columns[0]))print returned objectfacets=project1.compute_facets(facet.TextFacet(project1.columns[0])).facets[0]forkinsorted(facets.choices,key=lambdak:facets.choices[k].count,reverse=True):print(facets.choices[k].count,k)compute clusters on first column:project1.compute_clusters(project1.columns[0])apply rules from file to project:project1.apply_operations('duplicates-deletion.json')export project:project1.export(export_format='tsv')print the returned fileobject:print(project1.export(export_format='tsv').read())save the returned fileobject to file:withopen('export.tsv','wb')asf:f.write(project1.export(export_format='tsv').read())templating export (function was added in this fork, seeAdvanced Templatingabove):data=project1.export_templating(prefix='''{ "events" : [''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',rowSeparator=''',''',suffix='''] }''')print(data.read())print help screen with available commands (many more!):help(project1)example for custom commands:project1.do_json('get-rows')['total']delete project:project1.delete()See also:Jupyter notebook by Trevor Muñoz (2013-08-18):Programmatic Use of Open Refine to Facet and Cluster Names of 'Dishes' from NYPL's What's on the menu?Jupyter notebook by Tony Hirst (2019-01-09)Notebook demonstrating how to control OpenRefine via a Python client.Unitteststest_refine.pyandtest_tutorial.py(both importingrefinetest.py)OpenRefine APIin official OpenRefine wikiBinderfree to use on-demand server with Jupyter notebook, OpenRefine and Bashno registration needed, will start within a few minutesrestrictedto 2 GB RAM and server will be deleted after 10 minutes of inactivitybash_kernel demo notebookfor using the openrefine-client in a Linux Bash environmentpython2 demo notebookfor using the openrefine-client in a Python 2 environmentDevelopmentIf you would like to contribute to the Python client library please consider a pull request to the upstream repositoryrefine-client-py.TestsEnsure you have OpenRefine running (i.e. available athttp://localhost:3333). If necessary set the environment variablesOPENREFINE_HOSTandOPENREFINE_PORTto change the URL.The Python client library includes several unit tests.run all testspythonsetup.pytestrun subset test_facetpythonsetup.py--test-suitetests.test_facetThere is also a script that uses docker images to run the unit tests with different versions of OpenRefine.run tests on all OpenRefine versions (from 2.0 up to 3.4.1)./tests.sh-arun tests on tag 3.4.1./tests.sh-t3.4.1run tests on tag 3.4.1 interactively (pause before and after tests)./tests.sh-t3.4.1-irun tests on tags 3.4.1 and 2.7./tests.sh-t3.4.1-t2.7For Linux there are also functional tests for all command line options.run all functional tests on OpenRefine 3.4./tests-cli.sh3.4.1run all functional tests on OpenRefine 3.4 with one-file-executable./tests-cli.sh3.4.1openrefine-client_0-3-7_linuxDistributingNote to myself: When releasing a new version...Run functional testsforvin2.72.83.03.13.23.33.43.4.1;do./tests-cli.sh$vdoneMake final changes in Gitupdate versions (e.g. 0.3.7 und 0-3-7) inREADME.mdupdate version insetup.pycheck ifDockerfileneeds to be changedBuild executables with PyInstallerRun PyInstaller in Python 2 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with:Ubuntu 16.04 LTS (64-bit)macOS Sierra 10.12 (64-bit)Windows 7 (32-bit)One-file-executables will be available indist/.gitclonehttps://github.com/opencultureconsulting/openrefine-client.gitcdopenrefine-client
python2-mpipinstallpyinstaller--user
python2-mpipinstallurllib2_file--user
python2-mPyInstaller--onefilerefine.py--hidden-importgoogle.refine.__main__Run functional tests with Linux executableforvin2.72.83.03.13.23.33.43.4.1;do./tests-cli.sh$vopenrefine-client_0-3-7_linuxdoneCreate release in GitHubdraftrelease notesand attach one-file-executablesBuild package and upload to PyPIpython3setup.pysdistbdist_wheel
python3-mtwineuploaddist/*Update Docker containeradd new autobuild for release versiontrigger latest buildBump openrefine-client version in related projectsopenrefine-batch:openrefine-batch.shandopenrefine-batch-docker.shopenrefineder:postBuildCreditsPaul Makepeace, authorDavid Huynh, [initial cut](<http://markmail.org/message/jsxzlcu3gn6drtb7)Artfinder, inspirationFelix Lohmeier, extended the CLI featuresSome data used in the test suite has been used from publicly available sources:louisiana-elected-officials.csv: fromhttp://www.sos.louisiana.gov/tabid/136/Default.aspxus_economic_assistance.csv:"The Green Book"eli-lilly.csv:ProPublica's "Docs for Dollarsleading to aLilly Faculty PDFprocessed byDavid Huynh's ScraperWiki script