Package: codalab-yaml-validator
Codalab Yaml ValidatorCodalab Yaml Validator is a command line tool made to be used in conjunction with Codalab V2. It validates competition
bundles locally without having to upload to the server first. It can also be used to compare one bundle to another, and
show the differences between them. Functionally, this is aimed at comparing a bundle used to upload a competition with
a dump of that competition created by the server, used to point out any differences between the two. This can be used
both to validate that changeshave notbeen made during the upload process, and also to validate that any changes
made in the editor on the serverhavebeen accounted for in the dumps file.InstallationUsingpippipinstallcodalab_yaml_validatorUsageSingle Directory ValidationThis can be used to validate a folder or a .zip file.# Validating a foldervalidate_bundle/path/to/folder/# Validating a zip filevalidate_bundle/path/to/file.zipOutputFirst, the yaml filecompetition.yamlis run through an initial formatting validation. This is done using the expected
schema (provided below). If there are errors on this level, aValueErroris raised and the validation process stops.Example error messageTraceback
...
ValueError: 
Error validating data /.../competition.yaml with schema /.../site-packages/codalab_yaml_validator/schema.yaml
	tasks.0.index: Required field missingIf the first validation process is passed,Yaml file passed initial formatting testsis printed and a deeper validation
process begins. This verifies things like the same index is not used on multiple phases, or that the files provided for
this like images and scoring programs actually exist at the provided file path. In this process there are bothErrorsandWarnings. Errors will prevent a bundle from being valid, and thus cannot be uploaded to Codalab, while warnings
are not invalid bundles, but uploading the bundle may not produce the desired competition.ExampleWARNINGS:
- Task with index 0: If specifying a key, all other fields will be ignored on upload
ERRORS:
- Duplicate task index(es): [0]
- Task index: "1" on phase: "Example Phase Name" not present in tasks
- File for scoring_program - (path/to/scoring_program.zip) - not foundIf there are no errorsYaml bundle is validwill be printedBundle to Bundle Comparisonvalidate_bundle/path/to/bundle/one/path/to/bundle/twoJust as before, both directories and zip files are acceptable, and one can be compared to the other, i.e.,validate_bundle/path/to/zip.zip/path/to/folderBundles are each run through the single bundle validation before comparisons are made. If either bundle is invalid,
the comparisons will not be made and errors must be addressed. If no errors are present, comparison will begin.Note: This validation is run silently, so warnings will not be printed, nor will validity affirmations. The only
feedback that will be printed are errors to be addressed.If both bundles are valid, comparisons will be made. Because the competition editor on Codalab allows for changing every
value present in an upload bundle, and the dumps process may print things like Tasks in different orders than they were
uploaded in, there is no definitive way toknowwhich Task originated with which. This comparison process examines all
possible options and compares the ones that match the closest.For example, if the upload bundle looks like:# ...phases:-index:0name:Fast Phasedescription:Computing Pi Fasterstart:02-01-2019end:09-01-2019tasks:-1-index:1name:Slower Phasedescription:Computing Pistart:08-01-2018end:02-01-2019tasks:-0# ...And the dump bundle looks something like:# ...phases:-index:0name:Slow Phasedescription:Computing Pistart:08-01-2018end:02-01-2019tasks:-0-index:1name:Fast Phasedescription:Computing Pi Quicklystart:02-01-2019end:09-01-2019tasks:-1# ...The Comparison process can intelligently determine that index 0 in the upload bundle should be compared to index 1 in the
dump bundle, so that the most accurate account of differences can be given. This does have some limitations, especially
as the number of changes made in the editor increase, but it should seek to minimize the number of differences when
making comparisons. This process is the same for comparing tasks, solutions, leaderboards, and columns.Example OutputIn the case of the above yamls:$ validate_bundle /path/to/Archive/ /path/to/Dump.zip
Differences:

- Values on Phases index:1 in Archive and index:0 in Dump.zip do not match for key: name.
  - Archive = Slower Phase
  - Dump.zip = Slow Phase

- Values on Phases index:0 in Archive and index:1 in Dump.zip do not match for key: description.
  - Archive = Computing Pi Faster
  - Dump.zip = Computing Pi QuicklyLimitationsCodalab allows uploading things like scoring programs in unzipped directories and zips them itself during the
upload process. When a dump is created, these zipped directories are returned. Hashes are used to compare files like this
so the folder must be compressed and then hashed. The compression of this directory yields a different hash than its
already compressed counterpart, so these files must be validated manually.While a bundle using the same hierarchy as Codalab v1.5 is currently acceptable to upload to Codalab v2, its validation
is outside the scope of this tool. Bundle must be of the "Task Solution" style to be validated properly.EOFs on pages are changed in the process of storing their content as text on the server, so when a dump of that content
is created, even if the characters are the same, the contents of the file differ slightly so verifying these with a
hash is impossible. Again, these files will need to be checked manually. Because every page would be flagged as different,
pages are not checked for differences at all.SchemaUsed in conjunction withYamaleto validate the yaml formatting.title:str(min=8, max=255)image:str(max=1024)tasks:list(include('task'))solutions:list(include('solution'), required=False)pages:list(include('page'))phases:list(include('phase'))leaderboards:list(include('leaderboard'))---page:title:str(max=32)file:str(max=1024)phase:name:str(max=128)description:str(max=1024)index:int(max=99, required=False)max_submissions:int(required=False)max_submissions_per_day:int(required=False)execution_time_limit_ms:int(max=5184000, required=False)start:date()end:date(required=False)tasks:list(int())solutions:list(int(), required=False)file:name:str(max=128)description:str(max=1024)path:str(max=1024)task:# key or scoring_program is requiredindex:int()name:str(max=256, required=False)description:str(max=1024, required=False)ingestion_program:str(max=1024, required=False)ingestion_only_during_scoring:bool(required=False)input_data:str(max=1024, required=False)scoring_program:str(max=1024, required=False)reference_data:str(max=1024, required=False)key:regex(r'[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}', required=False)solution:index:int()name:str(max=256)description:str(max=1024, required=False)path:str(max=1024, required=False)key:regex(r'[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}', required=False)leaderboard:title:str(max=128)key:str(max=128)index:int()columns:list(include('leaderboard_column'))force_submission_to_leaderboard:bool(required=False)force_best_submission_to_leaderboard:bool(required=False)disallow_leaderboard_modifying:bool(required=False)leaderboard_column:title:str(max=128)key:str(max=128)index:int()computation:enum('avg', required=False)computation_indexes:str(required=False)sorting:enum('asc', 'desc', required=False)decimal_count:int(required=False)