Saravanan666/pypi-org huggingface dataset:
Package: oarepo-model-builder-cf
Custom fields plugin for model builderCustom fieldsCustom fields allow extending model at configuration time, with the same sources.
This plugin adds two types of custom fields - ones defined on an explicit element
(usually the "custom_fields" element is used, but any name can be used) and other
one inlining the custom fields directly to the main content.Currently there is no support for inlining custom fields to "metadata" element,
but this can be changed in the future.Model fileCustom fields are defined on the level of "model":model:use:inveniocustom-fields:-element:custom_fieldsconfig:TEST_CUSTOM_FIELDSproperties:metadata:properties:title:fulltextExplicit custom fieldsExplicit custom fields are defined using a pair ofelement nameandconfigvariable.
In the instance the custom fields will be separated into theelement name:# configfrominvenio_records_resources.services.custom_fields.textimportKeywordCFTEST_CUSTOM_FIELDS=[KeywordCF('blah')]// instance
{
    "metadata": {
        "title": "My Title"
    },
    "custom_fields": {                  // "element" from model
        "blah": "Custom field value"    // as defined in "TEST_CUSTOM_FIELDS" variable
    }
}Inline custom fieldsInline custom fields are placed on the root level of the instance, without an enclosing element. To define them, just leave theelementdefinition. Note that only 1 definition of inline CF is allowed.model:use:inveniocustom-fields:-config:INLINE_CFproperties:metadata:properties:title:fulltext# configfrominvenio_records_resources.services.custom_fields.textimportKeywordCFINLINE_CF=[KeywordCF('blah')]// instance
{
    "metadata": {
        "title": "My Title"
    },
    "blah": "Custom field value"    // as defined in "INLINE_CF" variable
}Custom fields nested in metadataCustom fields nested inside the metadata are not supported yet.Using custom fieldsCustom fields influence the jsonschema file (adding a non-checked object or allowing additional properties) and the mapping file (adding an empty mapping for CF).During deployment time, after the index is created and before data are poured in,
a specialized mapping for custom fields must be created. To do so, callinveniooarepocfinit
Package: oarepo-model-builder-drafts-files
OARepo model builder drafts filesPlugin adding support for working with files on models separating drafts and published records. Based on the invenio model.The settings can be customized in the model yaml file under the draft_files element placed under model.ApiThe files plugin provides an extension of api for model with drafts to work with files.
The api is taken from InvenioRDM, the docs arehere
Package: oarepo-model-builder-expansions
OARepo model builder expansionsPlugin for oarepo-model-builder to allow invenio expandable fields. Expandable fields allow saving fields of other referenced records in the referencing record object.Expandable fields are specified on model level as a list and the yaml for
single expandable field has following structure:field namepath to the field in the referencing recordreferenced keyslist of paths to the fields in the referenced recordserviceservice for the referenced record, used to retrieve the referenced
record to get the field valuesservice-aliasnot required, specify alias for the service in the previous optionpid-fieldnot required, specify the field in the referencing record holding the
pid of the referenced record.expandable-field-classnot required, specify the class of the expandable
field, by default it'soarepo_runtime.expansions.expandable_fields.ReferencedRecordExpandableFieldPlugin use case example is in tests.
Package: oarepo-model-builder-files
OARepo model builder filesPlugin adding support for working with files based on the invenio model.Files are represented as another ("file") record connected with the original parent one.
The plugin generates the file record and modifies the parent record to create connection with new file one.
The file record is specified under "files" attribute in the model yaml file, see example
in tests.The plugin runs the original model builder on the files model in "files" profile,
reusing a lot of the model builder code with different configuration, notably with different
base classes for record, service, resource and config classes.
To get an idea which code is reused, see entrypoints. For configuration changes, see model preprocessors.ApiThe files plugin provides an api for working with files.
The api is by default accessible at {original model url}/{base record id}/files.
The api is taken from InvenioRDM, the docs arehereExampleA simple record with associated files can be defined as this:record:properties:metadata:properties:title:type:fulltextstatus:type:keywordmodule:name:thesisuse:-inveniofiles:properties:metadata:properties:title:type:fulltextmodule:name:thesisuse:-invenio_filessettings:schema-server:'local://'Using the api, first an
instance of the model has to be created. Then an instance of the
associated file record. The file content is then uploaded in separate step and
finally the upload must be commited.
Package: oarepo-model-builder-nr
NR modelThis github project generates:nr-metadata- runtime support containing marshmallow, serializers, ... for NR metadata (documents and data)oarepo-model-builder-nr- a plugin for oarepo-model-builder to generateIncompatible changesNone yet. If you introduce any incompatible changes (vs previous major version),
enumerate them here and link the readme file from the previous major version
for reference.UsageSee theexamples/model.yamlfile for a skeleton of a repository
which uses nr metadata document model. An even easier way is to use thenrpcommand to generate a documents- or data- compatible repository.
See the same pages to get help about the model contents.BranchesBranches are always named "aa.bb" and denote the version of the contained metadata schema.Updating packagePackage needs to be updated whenever the version of oarepo-model-builder and
other builder plugins changes and influences the generated files.The package version is always in the form ofaa.bb.ccc, whereaa.bbis the version
of the metadata schema andcccis the version of the python package.To rebuild python packages:Create a development branch (named after the issue inside linear or github issue)Implement your changesIncrease the version number inversionfileRunbuild.sh(note - this has to be run after modification to theversionfile)Create a merge request to theaa.bbbranchCreate a new github release.Updating modelTo update the model version,start by creating a new branch with the nameaa.bb, whereaa.bbrepresents the new model version.For example, if the current version is2.0and your changes are minor,
the new version should be2.1, while major changes warrant
a version bump to3.0.Set this branch as the default branch on GitHub.Rename files inside themodeldirectory to include the correct versionChange the model version inside thebuild.shscriptMake your modifications to the files within the model directoryUpdate the content of theversionfile to reflect the new version asaa.bb.0Execute thebuild.shscript to ensure that everything builds correctlyPush your changes to GitHubMark the branch as the default branchCreate a new GitHub release.
Package: oarepo-model-builder-polymorphic
Polymorphic model builder pluginAnOARepo Model Builderplugin to generate
invenio sources for polymorphic models.Installationpipinstalloarepo-model-builder-polymorphicUsageWhat is polymorphic model?A polymorphic model is a dict that can be defined by multiple schemas.
Which schema is used depends on a value of a present field (that is shared
across all the schemas).Polymorphic model can be used, for example, to express inheritance:extension1{}:disc:{type:keyword}a:{type:keyword}extension2{}:disc:{type:keyword}b:{type:keyword}record:properties:a:type:polymorphicdiscriminator:discschemas:"1":{use:"/extension1"}"2":{use:"/extension2"}The following are valid instances:a:disc:"1"a:"blah"---a:disc:"2"b:"blah"Invalid instance:a:disc:"3"b:"blah"---a:disc:"1"a:"blah"b:"blah"If the "disc" field contains value "1", first extension schema will be used,
if it contains value "2", second schema will be used. Any other value is not
valid.
Package: oarepo-model-builder-relations
Relations plugin for model builderRelation data typeThe relation data type marks part of the metadata that will be filled with a related object.Example usage - part of dataset model:primary-article:type:relationname:articlemodel:articlekeys:['metadata.title','metadata.authors']The element "primary-article" will be connected to the article record. The record
class and properties will be taken from the "article" model
that is registered in "oarepo_models" entrypoint.When returned to the user/serialized to ES, it will take the title and
author fields from the related model and copy them into the primary-article field.It will automatically generate the following schema:will generate:// schema
{
    "primary-article": {
        "type": "object",
        "properties": {
            "id": {"type": "string"},
            "@v": {"type": "string"},
            "title": {},     // copied from registered article model
            "authors": {}    // copied from registered article model
        }
    }
}classRecord:# ...relations=MultiRelationsField(article=PIDRelation(# "name" from above"...primary-article",# path inside the dataset model"keys":['metadata.title','metadata.authors'],# copied keyspid_field:Article.pid# resolved model name and PID field))All arguments:primary-article:type:relationname:...model:...keys:[...]model-class:relation-classes:list:PIDListRelationnested:PIDNestedListRelationsingle:PIDRelationrelation-class:relation-args:attrs:[]keys:[]_value_key_suffix:id_clear_empty:truecache_key:value_check:pid_field:model-class.pidimports:-import:invenio_records_resources.records.system_fields.relations.PIDRelationalias:PIDRelationFieldDescriptionnameRelation name, will be used as param name inside RelationsFieldmodelName of the referenced model - from oarepo.models entrypoint or passed on commandline via --include parameterkeysArray of paths or dicts. If item is a path, that path will be copied from the referenced model. If it is dict it must contain the whole model definition.model-classClass representing the related modelrelation-classesIf the field is not an array and not nested inside an array, "single" is used. If the field is an array "list" is used. If the field is inside an array field, "nested" is usedrelation-classcan be used to override relation-classesrelation-argsA dictionary of arguments that will be passes to the relation classimportsYou can define your own imports/aliases here. The defaults are thise for list, nested and single relation classesInternal relationsSometimes it you might want to reference part of your document for indexing purposes etc. and not split the document into two records.
For these, internal relations can be used:properties:metadata:properties:obj{}:^id:anchor-objtest:keywordid:keywordOn object/array item, define the "id" field containing your "symbolic" name of the target of the relation.
Then the definition of the relation will look like:properties:metadata:properties:internal-ref:type:relationmodel:"#anchor-obj"keys:[id,test]Supported relationsSeereferrer.yamlfor a list of supported relations.
Package: oarepo-model-builder-requests
OARepo model builder requestsPlugin for oarepo-model-builder to allow specifying requests
tied to the base model, based on invenio requests.Requests are requests for an action concerning an instance of the model (called
topic in this context),
for example to publish the topic.The requests are specified as request types (each type has its own class).
Each request type has pre-specified actions from the invenio framework.
The plugin allows to customize the accept action.By default, along with changing the request status,
the generated accept action fetches the associated record. It's up to the
developer to decide what to do with it further.Requests can be added to the model through "requests" section
in the yaml file of the model. There can be any number of them.
Options in the yaml:moduleModule in which the request is stored. The type and action modules are derived from it but can override the default.typeSettings for the type class of the request.
Used if the user wants to use a custom request type class.moduleclassbasesimportsgenerateTrue by defualt, False is
typically used when user provides custom Type class.parent-actionsRequest type class specifies actions that can be performed by the request. By default the class imports its parent's possible actions, but this behavior can be customized if the parent class has different structure for some reason.actionsA list of customized actions by this type of request. Only the accept action is supported for now.moduleclassAnalogous to type class for the action. By default
it'sinvenio_requests.customizations.AcceptAction.basesimportsgenerateAnalogous to type class for the action.basesAnalogous to type class bases for the action.It also allows specifying the topic record resolver used. The resolver is tied to the topic class therefore it's generated on record level instead of the request.
Configurable options:moduleclassbasesimportsgeneratecustom-proxy-classThe resolver can use different proxy record resolution class. This is needed for resolving drafts because they don't have registered PIDs.ExamplesFor example, the requests plugin can be used for approving
a thesis. The accept action can be used for approval.
A simple yaml entry at the root level can be used:requests:approve-thesis:{}This generates all the basic files. The generated action does
the predefined changes in the default invenio base class, specifically
changes the request status, and provides a template for modifying
the associated thesis record (topic). The template fetches the record and
saves changes on it. What to do with the thesis during is up to the developer.
They have to replace this code with their own implementation.## todo - do something with the record# topic["status"] = "accepted"##Alternatively, they can define the accept action class on their own and import
it instead of generating.requests:approve-thesis:class:thesis.requests.ApproveActiongenerate:False
Package: oarepo-model-builder-tests
OARepo model builder testsPlugin for oarepo-model-builder to generate
test files and add test dependencies.
The record service and its rest api are covered for now. Tests read, write,
update, delete and search operations.The tests use automatically generated metadata in {model_name}/data/sample_data.yaml file to create records and upload them as fixtures.Resource test:test_get_item: Tries to use the api to retrieve non existing and existing record. Should fail on the first task and success on second.test_create: Tries to use the api to create a list of records. Also tries to use unauthorized client to do so, which should fail with http code 403. Then the test tries to retrieve the records and checks whether they contain the same metadata that they were uploaded with.test_listing: Tries a get call on the base url to retrieve all saved records. Checks whether the correct number of records is retrieved.test_update: Tries to update non-existing record. Then updates an existing record and checks whether the metadata are correctly updated and revision_id incremented.test_delete: Tries to delete non-existing record. Then tries to delete an existing one and checks whether the get call return 410 error http code. Also checks that unauthorized client can't delete.test_search: Finds all metadata fields generated in the test data. Then tries search based on values in the fields and checks whether at least one record with the same field value is returned. Tries search based on created time, with correct time, wrong time and as a facet.The service tests are analogous, they only bypass the api and use the expected service methods directly.Installationmodel.yamlmodel:plugins:packages:-oarepo-model-builder-testscommand linepipinstalloarepo-model-builder
pipinstalloarepo-model-builder-tests
Package: oarepo-model-builder-ui
UI model builder pluginAnOARepo Model Builderplugin to generate
user interface layout from model specification.Installationpipinstalloarepo-model-builder-uiUsageThe plugin adds support for UI annotation of the model file:title:type:keywordlabel.key:title.labellabel.cs:Názevlabel.en:Titlehelp.key:title.helphint.key:title.hintui:detail:<detail component name>edit:<edit component name>// any other property here - not validatedThe following files are generated:<package>
  +- model
    +- ui.json 
  +- templates
    +- <langcode>
      +- LC_MESSAGES
        +- messages.po
        +- messages.moi18nThepoandmofiles are standard gettext po files. The translation keys
are taken from the*.keyproperties. If keys are not specified, they are
generated from the property path (for example, metadata.creator.firstName.label ).UI layoutThe UI layout is generated as a convenience for other tools, for example foroarepo-cli.
It contains simplified layout of the model. Example:{"children":{"title":{"detail":"fulltext","input":"fulltext","help":"title.help","label":"title.label","hint":"title.hint"},"providers":{"detail":"array","input":"array","help":"providers.help","label":"providers.label","hint":"providers.hint","child":{"detail":"provider","input":"provider","children":{"name":{"detail":"keyword","input":"keyword","help":"providers/name.help","label":"providers/name.label","hint":"providers/name.hint"}}}}}}Fields:detail: component used for rendering the detail pageinput: component used for rendering inputhelp/label/hint: keys to gettext message catalogues
Package: oarepo-profiler
Installationpipinstalloarepo-profilerUsagechange the wsgi endpoint frominvenio_app.wsgi:applicationtooarepo_profiler:applicationto start profiling, addOAREPO_PROFILER_ENABLED=Truetoinvenio.cfg. Implicitly theprofsfiles
will be created in/tmp/oarepo-profiler. This location can be changed by settingOAREPO_PROFILER_PATHuse third party tools (such as snakeviz) to interpret the result of profiling
Package: oarepo-rdm-records
OARepo rdm records modelInstalationpipinstalloarepo-rdm-recordsUsageThe library provides a modifiedInvenioRDMRecordsdata
model for OARepo resource records.JSON SchemaAdd this package to your dependencies and use it via $ref in json
schema as"[server]/dataset-v2.0.0.json#/definitions/DataSet".
Elastic Search mapping is handled automatically via Eleastic Search templates.Usage example{"$schema":"http://json-schema.org/draft-07/schema#","type":"object","allOf":[{"$ref":"dataset-v2.0.0.json#/definitions/DataSet"}]}{"type":"object","properties":{"these":{"titles":{"cs":"neco","en":"something"}}}}Modified fields of data modeltitlesfully required oarepo multilingual data typedescriptionsfully optional oarepo multilingual data typesubjectssubjectis required oarepo multilingual data typelocationsdescriptionis optional oarepo multilingual data typelicenseslicenseis required oarepo multilingual data typeChangesVersion 2.0.0 (released 2020-08-25)Initial public release.Version 3.1.6Metadata field "Identifiers" now has all properties set as optional and new property "status" was added (for DOI request purposes).
Package: oarepo-s3-cli
oarepo-s3-cliOARepo command-line upload clientglobal options-e, --endpoint<url>OARepo HTTPS endpoint e.g.https://repo.example.org(required)-t, --token<string>upload auth token obtained from OARepo (required, can be alternatively specified in env.variable "TOKEN")-d, debug (default: False)-q, quiet (default: False)-n, --noninteractive (default: False)--helpcommandsupload ... upload fileresume ... resume interrupted uploadabort ... abort uploadcheck ... match sha256sum of local and uploaded filerevoke ... revoke supplied access tokenuploadcommand options-f, --file<filepath>file(s) for upload (repeatable, required)-k, --key<name>object key in S3 (default: basename of file)-p, --parallel<integer>(default: CPU count)resumecommand options-k, --key<name>object key in S3 (default: basename of file)-u, --uploadId<string>uploadId returned from upload  (required)-f, --file<filepath>file for upload (required)-p, --parallel<integer>number of parallel upload streams (default: CPU count)abortcommand options-k, --key<name>object key in S3 (default: basename of file)-u, --uploadId<string>uploadId returned from upload  (required)checkcommand options-f, --file<filepath>uploaded file for check (required)-k, --key<name>object key of uploaded file in S3 (default: basename of file)revokecommand optionsnone..
Copyright (C) 2021 CESNET.OARepo-S3-CLI is free software; you can redistribute it and/or
modify it under the terms of the MIT License; see LICENSE file for more
details.ChangesVersion 0.1.0 (released TBD)Initial public release.Version 0.1.1noninteractive mode added, resume call movedVersion 0.1.2setuptools ver.limitedREADME.mdcheck command impl.revoke command impl.error messages changedVersion 0.1.3CLI entry point addedexception handling changedVersion 0.1.4added support for nr-test variantVersion 0.1.5impl. --version optionVersion 0.1.6mon.timeout processing fixedtimer impl.Version 0.1.7check cmd optimized (chunk 8192, blake2b)Version 0.1.8check cmd paralelizedVersion 0.1.9nocheck flag impl.etag multipart checksums supp.presign req. grouping impl.presign req. divided into batchesVersion 0.1.10changed chunk_size calculation + max_parts 10240
Package: oarepo-search
OARepo-SearchOArepo module that added auxiliary search features.InstallationOARepo-Search is on PyPI so all you need is:$pipinstalloarepo-searchConfigurationSupported languagesIf you use multilingual fields (oarepo-multilingual) and want use
simple query for searching in the fields, you have to specify supported languages.OAREPO_SEARCH_SUPPORTED_LANGUAGES=["cs","en"]Search fieldsField settings for simple search (https://host.com/path/?q=). Configuration is a dictionary, where key is the
name of the
endpoint and value is a list of supported fields. If you want to use a multilingual field, you can use an asterisk instead of the language, which is automatically replaced by OAREPO_SEARCH_SUPPORTED_FIELDS.:warning:Simple query does not support nested fields: If you want search in nested fields you must use Lucene
query with field specification (e.g.: "creator.role.title.cs:Karel Čapek")OAREPO_SEARCH_FIELDS={"entrypoint_name":["title.*","creator"]}UsageQuery parserThe library provides an extension to the default parser provided by Invenio. The use of the query parser is
described in the module [invenio-records-rest](https://invenio-records-rest.readthedocs.io/en/latest/usage.
html#query-parser).The library can use a simple query as in invenio when you enter a search query in the form: q = , but this simple expression cannot search in nested fields. Therefore, the library supports queries in Lucene syntax with support for nested field search (eg "title: robot AND creator.name:Capek")Provide query parser from this library into your search_factory.frominvenio_records_rest.queryimportdefault_search_factoryfromoarepo_search.query_parsersimportquery_parserdefmy_search_factory(*args,**kwargs):returndefault_search_factory(*args,query_parser=query_parser,**kwargs)RECORDS_REST_ENDPOINTS={'recid':{# ...'search_factory_imp':my_search_factory,}}Copyright (C) 2021 CESNET.OARepo-Search is free software; you can redistribute it and/or
modify it under the terms of the MIT License; see LICENSE file for more
details.ChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a Changelog,
and this project adheres toSemantic Versioning.[Unreleased]AddedBetter explanation of the difference between the file ("CHANGELOG")
and its function "the change log".ChangedRefer to a "change log" instead of a "CHANGELOG" throughout the site
to differentiate between the file and the purpose of the file — the
logging of changes.RemovedRemove empty sections from CHANGELOG, they occupy too much space and
create too much noise in the file. People will have to assume that the
missing sections were intentionally left out because they contained no
notable changes.