Package: oarepo-validate
oarepo-validateOArepo Validate library for model-level matedata validationTable of ContentsInstallationUsageJSON schema validationCreate recordPUT / PATCH recordMarshmallow validationUsageWhat about marshmallow in loader?record-filesContextOutput of marshmallow validationValid dataInvalid dataMerging processSignalsSerializersInstallationpipinstalloarepo-validateUsageThe library provides mixins for enforcing json schema and marshmallow validation.JSON schema validationIf$schemais present on metadata, invenio performs a json schema validation inside
thevalidate()method. The problem is that$schemacan be set/removed via the REST
API. This means that an ill-written client can completely bypass the validation.To mitigate this issue, create your own Record implementation:fromoarepo_validateimportSchemaKeepingRecordMixinfrominvenio_recordsimportRecordclassMyRecord(SchemaKeepingRecordMixin,Record):ALLOWED_SCHEMAS=('records/record-v1.0.0.json','records/record-v2.0.0.json')PREFERRED_SCHEMA='records/record-v2.0.0.json'And register the record in REST endpoints in configuration:RECORD_PID='pid(recid,record_class="my:MyRecord")'RECORDS_REST_ENDPOINTS={'records':dict(pid_type='recid',pid_minter='recid',pid_fetcher='recid',record_class='my:MyRecord',item_route='/records/<{0}:pid_value>'.format(RECORD_PID),# ...)}Create recordWhen creating a new record, if$schemais not set,MyRecord.PREFERRED_SCHEMAis added
automatically. If$schemais set, it is validated againstMyRecord.ALLOWED_SCHEMASand an exception is raised if the schema is not present inALLOWED_SCHEMAS.PUT / PATCH recordBefore the result of the operation is committed,$schemais checked again.Marshmallow validationIn invenio, REST create operation use the following sequence:<flask>
<invenio_records_rest.views.RecordsListResource:post>
   <loader>
      <marshmallow>
   <permission factory>
   <pid minter>
   <record_class.create>
      <record.commit>
         <record.validate>REST PUT operation then uses:<flask>
<invenio_records_rest.views.RecordResource:put>
   <permission factory>
   <loader>
      <marshmallow>
   <record.update>
   <record.commit>
      <record.validate>REST PATCH operation:<flask>
<invenio_records_rest.views.RecordResource:put>
   <permission factory>
   <simple json loader>
   <record.patch>
   <record.commit>
      <record.validate>As you can see, if you place any validation code in loader's marshmallow, it is not executed.
An alternative is to have the validation code invalidateand handle all validations there.
This library does exactly this - it provides a record mixin that calls marshmallow schema'sloadmethod inside itsvalidatemethod.UsageCreate your own record and inherit from the mixin:fromoarepo_validateimportMarshmallowValidatedRecordMixinfrominvenio_recordsimportRecordfrommarshmallowimportSchema,fieldsclassTestSchema(Schema):name=fields.Str(required=True)classMyRecord(MarshmallowValidatedRecordMixin,Record):MARSHMALLOW_SCHEMA=TestSchemaDo not forget to register it as in the previous example.Now marshmallow schema will be processed before eachcommitmethod.What about marshmallow in loader?In most cases, marshmallow schema in loader can be removed and a simple json loader used instead.
However, if you need a custom processing of input data that is independent of validation,
you can keep the two marshmallows. To remove marshmallow loader and use a simple one,
setoarepo_validate.json_loaderas the record loader.RECORDS_REST_ENDPOINTS={'recid':dict(record_loaders={'application/json':'oarepo_validate:json_loader',},# ...)}A special case is when the marshmallow in loader already includes validation marshmallow rules.
Then you would want to use loader's marshmallow for create / replace and marshmallow in validation
only for patch operation (so that the same marshmallow rules are not called twice). To accomplish
this, set:classMyRecord(MarshmallowValidatedRecordMixin,Record):MARSHMALLOW_SCHEMA=TestSchemaVALIDATE_MARSHMALLOW=FalseVALIDATE_PATCH=TrueVALIDATE_MARSHMALLOWwill switch off marshmallow validation invalidatemethod andVALIDATE_PATCHwill switch on marshmallow validation inpatchmethod.record-filesBe careful with removing the loader when you useinvenio-record-files. Just using plain
json loader makes it possible to set_bucketand_filesdirectly which should be
disabled for security reasons (anyone might gain access to any file if he knows bucket and
object version of the file and has write access to any record).To fix this, set:fromoarepo_validateimportFilesKeepingRecordMixinRECORDS_REST_ENDPOINTS={'recid':dict(record_loaders={'application/json':'oarepo_validate:json_loader',},# ...)}classMyRecord(FilesKeepingRecordMixin,...):...The loader will strip_bucketand_filesfrom the payload and the mixin
will make sure that the files can not be removed withputor replaced withpatchoperation.ContextMarshmallow validation is called with a context, that is filled with:recordpidif it is knownAny**kwargspassed toRecord.createorRecord.commitOutput of marshmallow validationValid dataThe marshmallow loader produces validated data. Be default, the validated data are merged into
the record. The rationale for this is that the validation might be used to replace content
(include referenced content, etc). To have a consistent processing, the schema must be idempotent,
that isschema(schema(input)) == schema(input).To prevent this behaviour, setMERGE_WITH_VALIDATEDtoFalseon your record class.Invalid dataEven in the case the data are invalid, marshmallow validation might still return a partially
valid output (inValidationError.valid_data). This library merges the valid data into
the record's metadata. This behaviour can be switched off by settingMERGE_WITH_VALIDATED_ERROR = Falseon your record class.Merging processThe merging process is recursive and is designed to preserve values in the record's metadata
if they are not present in the validated metadata. This means:lists are merged item-wise. If the list in the record is longer than in validated data,
extra items are kept.dictionaries are merged based on the same key. If a key is both in the record and in validated
data, the respective values are merged recursively and the result used. Extra keys from
validated data are copied into the record's metadata and keys present in record's metadata
and ommited from validated data are kept.Sometimes it might be necessary to prevent this merging. The library provides:Keep(value)- if an instance ofKeepclass is returned, no merging is performed
and the value is used as is - that is, it will overwrite anything in the recordDELETED- if this constant is returned, the item will be deleted - if it is a part of
an array, the corresponding array item in record's metadata is deleted. If it is a value
of a key in the dictionary, the corresponding key in record's metadata is deleted.SignalsThe library provides the following signals:before_marshmallow_validate=signal('oarepo_before_marshmallow_validate')"""Signal invoked before record metadata are validated (loaded by marshmallow schema)inside Record.validate:param source:  the record being validated:param record:  the record being validated:param context: marshmallow context:param **kwargs: kwargs passed to Record.create or Record.commit (or Record.validate)"""after_marshmallow_validate=signal('oarepo_after_marshmallow_validate')"""Signal invoked after record metadata are validated (loaded by marshmallow schema)inside Record.validate:param source:  the record being validated:param record:  the record that was successfully validated:param context: marshmallow context:param result:  result of load that will be used to update record's metadata.Signal handler can modify it. In case of validation exception the result is None.:param error:   Exception raised when validating. None if validation has been successful:param **kwargs: kwargs passed to Record.create or Record.commit (or Record.validate)"""SerializersIfmarhsmallow.dumpis not required for metadata serialization,oarepo_validate.json_search, oarepo_validate.json_responseare faster replacements for marshmallow-based serializers:RECORDS_REST_ENDPOINTS={'recid':dict(record_serializers={'application/json':'oarepo_validate:json_response',},search_serializers={'application/json':'oarepo_validate:json_search',})}ChangesVersion 1.2.3 (released 2020-08-30)Handling pid field in search hit serializationVersion 1.2.2 (released 2020-08-25)Handling pid field in record serializationVersion 1.2.1 (released 2020-08-25)Keeping schema inRecord.__init__(useful mostly for tests)Version 1.2.0 (released 2020-08-25)Added marshmallow-less loaders and serializersVersion 1.1.0 (released 2020-08-18)Added before and after validation signals.Version 1.0.0 (released 2020-08-16)Initial public release.