Package: dlai-grader
graderAutomatic grading for DLAI courses. Designed to be compatible with Coursera's grading requirements.RequirementsTo use this library you will need to have Python, Docker andcoursera-autograderinstalled in your pc.InstallationYou can use pip to install it!pipinstalldlai_graderHow to use itInitialize a graderTo start building your gradercdinto the directory where you will be working and use the following command:dlai_grader--initThis will ask you for:Name of the course (abbreviation is recommended)Number of the courseNumber of the weekVersion of the grader (defaults to 1 but can be 2 or any other number)This will generate a file system tree like this:.
└── grader (directory you invoked the command from)
    ├── data/               -> To store datasets (csv, TF Datasets, etc).
    ├── learner/            -> The learner facing version will be generated here.
    ├── solution/           -> Place solution.ipynb here
    ├── submission/         -> Necessary only in debug mode (no need to place anything here).
    ├── mount/              -> This mocks the bind mount that coursera will attach to the container. Should contain submission.ipynb or other file required for grading.
    ├── .conf               -> Configuration variables.
    ├── Dockerfile          -> Uses frolvlad/alpine-miniconda3:python3.7 as base image.
    ├── Makefile            -> Useful commands.
    ├── requirements.txt    -> Python dependencies.
    ├── entry.py            -> Entrypoint of the grader.
    └── grader.py           -> Grading logic.Placing the solution and submissionNow that you have the layout of the grader you will need to place the solution of the assignment within thesolution/directory.This file must be namedsolution.ipynb.A good starting point is to use the solution to create the first iteration of the grader. To do this place the solution within themount/directory andrename it tosubmission.ipynb. You can use themake submit-solutioncommand to do this.Your filesystem tree should look like this:.
└── grader
    ├── ... 
    ├── solution/
    │    └── solution.ipynb
    ├── mount/ 
    │    └── submission.ipynb
    └── ...Note that the grader can be used to grade files other than Jupyter notebooks. If this is the case you can leave thesolution/directory empty and place the file to grade withinmount/. This file can be anything (.h5, .tar.gz, .zip, etc) the only requirement is that it should match the name of the file to submit in the coursera programming item.Add versioning to the notebookWhy is this useful?We have seen many learners facing issues when submitting their assignments because coursera does not show them the latest version. To address this, a good alternative it to always check that the submission of the learner is up to date and if not, tell them how they can upgrade to the latest version.To ensure that a submission is compatible with a particular version of the grader the versioning feature has been created.Within the.conffile you will find the current version of the grader under the variableGRADER_VERSION.To compare against this version the submission notebook must include a variable calledgrader_versionin its metadata. To add this variable to thesubmission.ipynbfile you can use themake versioningcommand (this is just a wrapper of thedlai_grader --versioningcommand). This will add the variable matching the same version as the one found in the.conffile.Upgrading the grader and notebook versionAfter a refactor with breaking changes to the grader it is a good idea to upgrade the version to a newer one. You can do this be using themake upgradecommand. This will add 1 to the current version in the.conffile and in the notebook.Tagging graded cellsYou might decide to filter out cells created by learners (or other ones such as the ones that train models). If you take this approach you can add a tag to each cell's metadata and then filter out the ones that don't have this tag .If you wish to add thegradedtag to each cell in thesubmission.ipynbnotebook you can use themake tagcommand.Important note for make commandsNote that themake tag,make upgradeandmake versioningcommands change only themount/submission.ipynbfile.Adding Python dependenciesThe next step is to include all the necessary Python dependencies. For this add them to therequirements.txtfile. By default onlydlai_graderis included.Building the gradingNotice that two blank Python files were created during theinitstep. These aregrader.pyandentry.py. It might be odd to have two separate files if ony could do the trick.However it is usually best to separate the grading logic from the entrypoint of the application, as the names suggest, these should be placed withingrader.pyandentry.py, respectively.Building the Docker imageEvery time you make changes to any file that the grader is dependant-on, you should rebuild the Docker image used for grading. You can do this by using themake buildcommand.GradingTo grade, simply use themake gradecommand. This will spin up the coursera-autograder tool to simulate Coursera's grading environment.DebuggingSome times when using this command you will be presented with a not so descriptive error by coursera-autograder such as "Problem when running command. Sorry!". This happens because this tool does not transparently exposes the errors.When you face this error you can enter debug mode by using themake debugcommand. This will spin up a container using the image you created and configuring it in the same way as coursera-autograder would.Within the container you have a command line where you can test your code directly. To do this, export the partid you are currently working on as an environment variable.export partId=XxXxwill do the trick. And then use the entrypoint of the application by runningpython entry.py.By doing this you can actually see the error messages that are being generated and another nice feature of this mode is that you don't need to rebuild the image, you can simply editgrading.pyand runpython entry.pyto try out the grader.Building by ExampleTo better ilustrate how to use the library, an example will be used. Theassignmentfor the first week of the first course of the Tensorflow 1 specialization will be used.Trimming the markdown and including the solution, the assignment looks like this:importtensorflowastfimportnumpyasnp# GRADED FUNCTION: house_modeldefhouse_model():### START CODE HERE# Define input and output tensors with the values for houses with 1 up to 6 bedrooms# Hint: Remember to explictly set the dtype as floatxs=np.array([1.0,2.0,3.0,4.0,5.0,6.0],dtype=float)ys=np.array([1.0,1.5,2.0,2.5,3.0,3.5],dtype=float)# Define your model (should be a model with 1 dense layer and 1 unit)model=tf.keras.Sequential([tf.keras.layers.Dense(units=1,input_shape=[1])])# Compile your model# Set the optimizer to Stochastic Gradient Descent# and use Mean Squared Error as the loss functionmodel.compile(optimizer='sgd',loss='mean_squared_error')# @REPLACE model.compile(optimizer=None, loss=None)# Train your model for 1000 epochs by feeding the i/o tensorsmodel.fit(xs,ys,epochs=1000)# @REPLACE model.fit(None, None, epochs=None)### END CODE HEREreturnmodel# Get your trained modelmodel=house_model()new_y=7.0prediction=model.predict([new_y])[0]print(prediction)Download the assignment