Package: manytask-checker
Manytask CheckerScript to test students' solutions withmanytaskintegrationKey features:git changes detectionextension for different languagessandbox executionmanytaskintegrationPlease refer to themanytaskdocumentation first to understand the drillHow it worksThecheckerlib is a relatively small cli script aiming to run tests in gitlab runner and push results tomanytask.The fullcheckerandmanytasksetup roughly looks as followsself-hostedgitlabinstance - storing repos with assignments and students' repoprivate repo - a repository with tasks, public and private tests, gold solutions, ect.public repo - a repository available to students with tasks and solution templatesstudents' group - the group wheremanytaskwill create repositories for studentseach students' repo - fork from public repogitlab runners- place where students' solutions likely to be testedcheckerscript - some script to test students' solutions and push scores/grades to themanytaskmanytaskinstance - web application managing students' grades (in google sheet) and deadlines (web page)The flow for checking students' solution looks like:Student push his solution to a gitlab repogitlab-ci runs separate docker in gitlab-runnergitlab-ci runs this script with some parametersthe script detect the latest changes (via git) and select tasks to checkthe tasks forwarded totesterand it returns obtained scoresthe script push student scores to the manytask(additionally script can check ground-truth solutions, export new tasks etc)UsagePre requirementsmanytaskweb app
Currently, this lib is integrated with manytaskonly,
so you need it to be set up first, see installation instructions in manytask repo.gitlab with access to greate groups, users and add runnersThis pre-requirement for manytask; See manytask installation instructions for more infoCreated and testedtesterfor your course/languagePreparationsObtain service keys for this script to operatemanytask tester token you set up when run itgitlab service user to operate with your repositories(better to create a new one)Create gitlab repositories layoutCreate private repository with tasks, public and private tests and ground-truth solution;Choose one of the suitable layouts (seedriver.py)Grant access to your service accountCreate public empty repositoryGrant access to your service accountCreate private (!) group for students repositories(You have already done it if you set up manytask)Grant access to your service accountEdit config files in repository.course.yml- main endpoints config(see.course.yml example).deadlines.yml- task deadlines
(see.deadlines.yml example).gitlab-ci.yml- set up gitlab ci pipeline to test students tasks.releaser-ci.yml- set up gitlab ci pipeline to test new added tasks and build dockersSetup dockers with env ready for testing, it's convenient to have 2 dockers:base.docker- base docker to build and test students solutions, install lib heretestenv.docker- docker on top of base docker, to save tasks and testsStructureCourseCourseConfigManage course configuration. Wrapper around.course.ymlfile.CourseScheduleManage course deadlines. Wrapper around.deadlines.ymlfile.CourseDriverManage mapping of the Course to the Filesystem. (e.g. map Task to folders with tests and source files)Available layouts are (seedriver.py):flat- all tasks in root folder of the repogroups- each group has its own folderTestingExecutoris object to run commands with some isolation level.Available modes are:sandbox- separate process (clean env variables, nouser/nogroup, disabled network)docker- TODOTesteris object which can test single task: copy files, build, test it, cleanup.Tester is extendable for each course/language. Now available:pythonDevelopingInstallationCreate venvpython-mvenv.venvsource.venv/bin/activateInstall lib in dev mode(.venv)$pipinstall-U--editable.[test]# .\[test\] in zshRunning tests and linterspytest.--cpp--pythonruffchecker
mypychecker
isort--check.Adding a new language testerIn order to add a new language to the test system you need to make a pull request.Add a new tester inchecker/testers(seepython.pyas example)Updatetester.pycreatemethod to run your testerWrite tests for a new tester in./tests/testers