Repository: oppia/oppia

Issue:
Update docstrings in the Python backend code to follow the style guide.
We've been a bit lax about adding docstrings to code in the backend. Instead, we should be following the style guide: https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments -- docstrings should define args, return values, exceptions, etc. The aim of this issue is to ensure that the backend code is properly documented; we will tackle this on a file-by-file basis.

Note: if you're a new contributor, this issue is a particularly good introductory one to tackle, since it will require reading, understanding and documenting the purpose of a file in the codebase, and help you get acquainted with how the Oppia backend is structured. It's important that you're able to communicate and write clearly, though!

Some useful tips:
-  Please follow our [style guide](https://github.com/oppia/oppia/wiki/Coding-style-guide#general) (and also the [Google Python style guide](https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments)) when writing docstrings. Make sure to follow the patterns established there as closely as possible, including spacing, capitalization and punctuation. Also, it's a good idea to look at some of the existing files (see below) to get a sense of the correct format -- we recommend following this closely in order to maintain uniformity across the codebase.
- If applicable, make sure that the docstrings for all functions correctly specify the args, return values, etc. If a function has no args, you can omit the Args section; if it has no return value, you can omit the Returns section.
- If a docstring is not present in the code, or the existing one is too vague for you to understand what's going on just by looking at it, then it should be rewritten to be more clear. You'll often need to read the code as well, and do a grep on the callsites to see how it's being used, in order to get a clear understanding of what it's doing. This is good practice for learning how to navigate the codebase, and is an intended part of this project.
- If you feel that parts of the codebase are unclear or incorrectly factored, please feel free to bring this up! The whole point of docstrings is to improve code health and accessibility for new developers, so if you spot other things that can be done in service of that aim, we should also look into doing that.

**Completed:**
- [x] core.controllers.base @shaz13 
- [x] core.controllers.cron @saeedjassani
- [x] core.controllers.reader @96mohitm
- [x] core.domain.activity_domain @andromfins
- [x] core.domain.activity_services @souravsingh
- [x] core.domain.classifier_services @andromfins 
- [x] core.domain.collection_domain @domevit
- [x] core.domain.collection_services @kahkhang 
- [x] core.domain.config_services @vibhor98 
- [x] core.domain.email_manager @Oishikatta 
- [x] core.domain.exp_domain @domevit
- [x] core.domain.exp_services @DubeySandeep
- [x] core.domain.feedback_domain @terrameijar 
- [x] core.domain.feedback_jobs_continuous
- [x] core.domain.feedback_services @andimeo
- [x] core.domain.fs_domain @kartik-25 
- [x] core.domain.html_cleaner @seanlip
- [x] core.domain.param_domain @rohan-gulati
- [x] core.domain.rating_services @BojOnTheBeat 
- [x] core.domain.rights_manager @ayejay
- [x] core.domain.stats_domain @Arunabh98
- [x] core.domain.stats_jobs_continuous @Arunabh98 
- [x] core.domain.stats_jobs_one_off @zpuller
- [x] core.domain.stats_services @manoj-makkuboy 
- [x] core.domain.subscription_services @vibhor98 
- [x] core.domain.summary_services @domevit 
- [x] core.domain.user_jobs_continuous @shubha1593 
- [x] core.domain.user_services @Nalinc 
- [x] core.domain.value_generators_domain @meetsha 
- [x] core.jobs @brianrodri
- [x] core.platform.app_identity.gae_app_identity_services @ubhisat
- [x] core.platform.models @pranavsid98 
- [x] core.platform.transactions.gae_transaction_services @ubhisat
- [x] core.storage.base_model.gae_models @vojtechjelinek
- [x] core.storage.collection.gae_models @kaggis
- [x] core.storage.email.gae_models @rajan-garg
- [x] core.storage.exploration.gae_models @kirankonduru
- [x] core.storage.feedback.gae_models @naveenshukla
- [x] core.storage.file.gae_models @Ian91 
- [x] core.storage.job.gae_models @sasacocic
- [x] core.storage.statistics.gae_models @abhishekarya286
- [x] core.storage.user.gae_models @Anthony-Alridge 
- [x] core.tests.test_util_jobs @seanlip
- [x] core.tests.test_utils @vibhor98
- [x] utils @yashLadha 


Discussion:
@seanlip I would like to work in the issue.

Hi @souravsingh, do you think you could finish up #2051 first, please? Thanks! 

@seanlip I would like to work on this issue.

Hi @andromfins -- now that you've had a go at writing docstrings for a single class, would you mind picking one or two other files and doing the same thing for those? Let me know which ones you pick, so that I can mark you off against them.

Thanks!

Hi, i'm working on **core.domain.classifier_services**.

OK, thanks @andromfins! 

@seanlip Can I take a stab at other file to add docstrings?

Yep, sounds good. Could you take core.domain.html_cleaner? Thanks!

@andromfins -- are you still working on this?

  @seanlip Yes!

I would like to work on `core.domain.stats_domain`

@Arunabh98 assigned you on `core.domain.stats_domain`

Almost completed `core.domain.stats_domain`. Now working on `core.domain.stats_jobs_continuous`

hi @seanlip 
I am working on this issue.
I am writing the docstrings for /core/storage/collection/gae_models.py as i was assigned as a starting project.

@seanlip can I work on core.controllers.cron?
Yep, sounds good! I've updated the list.
@seanlip I'd like to work on core.storage.job.gae_models.
Sounds good, thanks @sasacocic! I've updated the list. 
@seanlip I'd like to work on core.platform.models. Thanks

@seanlip I'd like to work on core.domain.stats_services
Sounds good, thanks @arjunmahishi! I've assigned you. 
@seanlip I'd like to work on core.domain.summary_services
Sounds good, thanks! I've assigned you.
@seanlip Hello, i would work on core.domain.config_domain if possible.
@tvallois I have assigned you to that. Thanks
@seanlip I can do core.domain.rights_manager.py
Sounds good @fodra, I've assigned you. Thanks! 
@seanlip I would like to take _core.tests.test_util_jobs_
Sounds great, I've assigned you. Thanks @terrameijar! 
@seanlip could i work on  core.storage.statistics.gae_models?

I'd like to contribute to this, could you please assign me one of the sub tasks?
Hi @fsam95, welcome to Oppia! Could you please follow the instructions [here](https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#setting-things-up) first?

@cosmicAsymmetry The list of tasks that are yet to be done is in the "Not started" section. Would you be up for taking a look at the corresponding files, and picking one that looks approachable? Note that test_utils has some prior work that you might be able to extend, though it's slightly unusual because the functions within it are meant mainly for supporting backend testing (and are therefore not directly user-facing). Thanks!
I've followed the instructions :). 
Thanks @fsam95! Assigned you. 
@seanlip I want to take one of the files but it looks like all the files in "Not started" section have been documented? Can you assign me to one which has not been taken? Thanks! :)
Hi @zhengxit -- actually, all those files still need work, I think, be it improving the existing docstrings, writing new docstrings for methods/functions which don't have them, or making sure that everything follows the correct style. So, please feel free to pick one of them!
(P.S. If you see a particular file in the "Not started" section that you're not sure what to do with, just ask.)
@seanlip I would like to take **core.controllers.base**

Assigned. Thanks @shaz13! 
I'm starting work on **core.jobs**.
@seanlip Please assign me an issue.I also want to contribute.
can I take #2394 Docstrings for controllers/base.py
@seanlip I am taking core.domain.stats_services
OK, thanks! Assigned you.
hey...I am working on core.domain.stats_services....I can see previous docstrings in it.So, what exactly is incomplete in this project.
Hello @arjunsuri, you can look at the previous PRs' ( alike #3448, #2459 ) to get an idea about expected format. I looked into the file [core.domain.stats_services. ](https://github.com/oppia/oppia/blob/develop/core/domain/stats_services.py) there are still few functions missing Args, Return information which you should document in your PR. If you still have any doubts feel free to ping us in the Gitter channel. 
I would like to work on `core.domain.value_generators_domain`
Sounds good, thanks @meetsha. I've assigned you! 

Resolving patch:
diff --git a/core/jobs.py b/core/jobs.py
--- a/core/jobs.py
+++ b/core/jobs.py
@@ -92,10 +92,12 @@ class BaseJobManager(object):
 
     Note that the enqueue(), register_start(), register_completion(),
     register_failure() and cancel() methods in this class batch the following
-    operations: (a) pre- and post-hooks, (b) updating the status of the job in
-    the datastore, and (c) actually performing the operation. Each entire batch
-    is not run in a transaction, but subclasses can still perform (a) or (c)
-    transactionally if they wish to.
+    operations:
+        (a) Running pre- and post-hooks
+        (b) Updating the status of the job in the datastore
+        (c) Performing the operation.
+    Each entire batch is not run in a transaction, but subclasses can still
+    perform (a) or (c) transactionally if they wish to.
     """
     @classmethod
     def _is_abstract(cls):
@@ -103,7 +105,15 @@ def _is_abstract(cls):
 
     @classmethod
     def create_new(cls):
-        """Creates a new job of this class type. Returns the id of this job."""
+        """Creates a new job of this class type.
+
+        Returns:
+            str. The unique id of this job.
+
+        Raises:
+            Exception: This method (instead of a subclass method) was directly
+                used to create a new job.
+        """
         if cls._is_abstract():
             raise Exception(
                 'Tried to directly create a job using the abstract base '
@@ -118,7 +128,13 @@ def _create_new_job():
 
     @classmethod
     def enqueue(cls, job_id, additional_job_params=None):
-        """Marks a job as queued and adds it to a queue for processing."""
+        """Marks a job as queued and adds it to a queue for processing.
+
+        Args:
+            job_id: str. The ID of the job to enqueue.
+            additional_job_params: dict(str : *) or None. Additional parameters
+                for the job.
+        """
         # Ensure that preconditions are met.
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_valid_transition(
@@ -138,6 +154,12 @@ def enqueue(cls, job_id, additional_job_params=None):
 
     @classmethod
     def register_start(cls, job_id, metadata=None):
+        """Marks a job as started.
+
+        Args:
+            job_id: str. The ID of the job to start.
+            metadata: dict(str : *) or None. Additional metadata of the job.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_valid_transition(
             job_id, model.status_code, STATUS_CODE_STARTED)
@@ -154,7 +176,12 @@ def register_start(cls, job_id, metadata=None):
 
     @classmethod
     def register_completion(cls, job_id, output_list):
-        """Marks a job as completed."""
+        """Marks a job as completed.
+
+        Args:
+            job_id: str. The ID of the job to complete.
+            output_list: list(object). The output produced by the job.
+        """
         _MAX_OUTPUT_LENGTH_CHARS = 900000
 
         # Ensure that preconditions are met.
@@ -197,7 +224,12 @@ def register_completion(cls, job_id, output_list):
 
     @classmethod
     def register_failure(cls, job_id, error):
-        """Marks a job as failed."""
+        """Marks a job as failed.
+
+        Args:
+            job_id: str. The ID of the job to fail.
+            error: str. The error raised by the job.
+        """
         # Ensure that preconditions are met.
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_valid_transition(
@@ -213,6 +245,12 @@ def register_failure(cls, job_id, error):
 
     @classmethod
     def cancel(cls, job_id, user_id):
+        """Marks a job as canceled.
+
+        Args:
+            job_id: str. The ID of the job to cancel.
+            user_id: str. The id of the user who cancelled the job.
+        """
         # Ensure that preconditions are met.
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_valid_transition(
@@ -233,19 +271,39 @@ def cancel(cls, job_id, user_id):
 
     @classmethod
     def is_active(cls, job_id):
+        """Returns whether the job is still active.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            bool. Whether the job is active or not.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.status_code in [STATUS_CODE_QUEUED, STATUS_CODE_STARTED]
 
     @classmethod
     def has_finished(cls, job_id):
+        """Returns whether the job has finished.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            bool. Whether the job has finished or not.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.status_code in [STATUS_CODE_COMPLETED, STATUS_CODE_FAILED]
 
     @classmethod
     def cancel_all_unfinished_jobs(cls, user_id):
-        """Cancel all queued or started jobs of this job type."""
+        """Cancels all queued or started jobs of this job type.
+
+        Args:
+            user_id: str. The id of the user who is cancelling the jobs.
+        """
         unfinished_job_models = job_models.JobModel.get_unfinished_jobs(
             cls.__name__)
         for model in unfinished_job_models:
@@ -255,49 +313,108 @@ def cancel_all_unfinished_jobs(cls, user_id):
     def _real_enqueue(cls, job_id, additional_job_params):
         """Does the actual work of enqueueing a job for deferred execution.
 
-        Must be implemented by subclasses.
+        Args:
+            job_id: str. The ID of the job to enqueue.
+            additional_job_params: dict(str : *). Additional parameters on jobs.
         """
         raise NotImplementedError(
             'Subclasses of BaseJobManager should implement _real_enqueue().')
 
     @classmethod
     def get_status_code(cls, job_id):
+        """Returns the status code of the job.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            str. Status code of the job.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.status_code
 
     @classmethod
     def get_time_queued_msec(cls, job_id):
+        """Returns the time the job got queued.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            float. The time the job got queued in milliseconds after the Epoch.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.time_queued_msec
 
     @classmethod
     def get_time_started_msec(cls, job_id):
+        """Returns the time the job got started.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            float. The time the job got started in milliseconds after the Epoch.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.time_started_msec
 
     @classmethod
     def get_time_finished_msec(cls, job_id):
+        """Returns the time the job got finished.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            float. The time the job got finished in milliseconds after the
+                Epoch.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.time_finished_msec
 
     @classmethod
     def get_metadata(cls, job_id):
+        """Returns the metadata of the job.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            dict(str : *). The metadata of the job.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.metadata
 
     @classmethod
     def get_output(cls, job_id):
+        """Returns the output of the job.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            *. The output of the job.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.output
 
     @classmethod
     def get_error(cls, job_id):
+        """Returns the error encountered by the job.
+
+        Args:
+            job_id: str. The ID of the job to query.
+
+        Returns:
+            str. Describes the error encountered by the job.
+        """
         model = job_models.JobModel.get(job_id, strict=True)
         cls._require_correct_job_type(model.job_type)
         return model.error
@@ -305,6 +422,16 @@ def get_error(cls, job_id):
     @classmethod
     def _require_valid_transition(
             cls, job_id, old_status_code, new_status_code):
+        """Asserts that the transition of the job status code is valid.
+
+        Args:
+            job_id: str. The ID of the job to query.
+            old_status_code: str. Old status code.
+            new_status_code: str. New status code.
+
+        Raises:
+            Exception: The given status code change is invalid.
+        """
         valid_new_status_codes = VALID_STATUS_CODE_TRANSITIONS[old_status_code]
         if new_status_code not in valid_new_status_codes:
             raise Exception(
@@ -313,6 +440,14 @@ def _require_valid_transition(
 
     @classmethod
     def _require_correct_job_type(cls, job_type):
+        """Returns validity of given job type.
+
+        Args:
+            job_type: str. Name of a job class.
+
+        Raises:
+            Exception: The given job type is incorrect.
+        """
         if job_type != cls.__name__:
             raise Exception(
                 'Invalid job type %s for class %s' % (job_type, cls.__name__))
@@ -356,13 +491,22 @@ class BaseDeferredJobManager(BaseJobManager):
     def _run(cls, additional_job_params):
         """Function that performs the main business logic of the job.
 
-        Needs to be implemented by subclasses.
+        Args:
+            additional_job_params: dict(str : *). Additional parameters on jobs.
         """
         raise NotImplementedError
 
     @classmethod
     def _run_job(cls, job_id, additional_job_params):
-        """Starts the job."""
+        """Starts the job.
+
+        Args:
+            job_id: str. The ID of the job to run.
+            additional_job_params: dict(str : *). Additional parameters on job.
+
+        Raises:
+            PermanentTaskFailure: No further work can be scheduled.
+        """
         logging.info(
             'Job %s started at %s' %
             (job_id, utils.get_current_time_in_millisecs()))
@@ -393,9 +537,9 @@ def _real_enqueue(cls, job_id, additional_job_params):
         """Puts the job in the task queue.
 
         Args:
-        - job_id: str, the id of the job.
-        - additional_job_params: dict of additional params to pass into the
-            job's _run() method.
+            job_id: str. The ID of the job to enqueue.
+            additional_job_params: dict(str : *). Additional params to pass into
+                the job's _run() method.
         """
         taskqueue_services.defer(cls._run_job, job_id, additional_job_params)
 
@@ -403,6 +547,20 @@ def _real_enqueue(cls, job_id, additional_job_params):
 class MapReduceJobPipeline(base_handler.PipelineBase):
 
     def run(self, job_id, job_class_str, kwargs):
+        """Returns a coroutine which runs the job pipeline and stores results.
+
+        Args:
+            job_id: str. The ID of the job to run.
+            job_class_str: str. Should uniquely identify each type of job.
+            kwargs: dict(str : object). Extra arguments used to build the
+                MapreducePipeline.
+
+        Yields:
+            MapreducePipeline. Ready to start processing. Expects the output of
+                that pipeline to be sent back.
+            StoreMapReduceResults. Will be constructed with whatever output the
+                caller sends back to the coroutine.
+        """
         job_class = mapreduce_util.for_name(job_class_str)
         job_class.register_start(job_id, metadata={
             job_class._OUTPUT_KEY_ROOT_PIPELINE_ID: self.root_pipeline_id  # pylint: disable=protected-access
@@ -413,7 +571,7 @@ def run(self, job_id, job_class_str, kwargs):
         yield StoreMapReduceResults(job_id, job_class_str, output)
 
     def finalized(self):
-        # Suppress the default Pipeline behavior of sending email.
+        """Suppresses the default pipeline behavior of sending email."""
         # TODO(sll): Should mark-as-done be here instead?
         pass
 
@@ -421,6 +579,13 @@ def finalized(self):
 class StoreMapReduceResults(base_handler.PipelineBase):
 
     def run(self, job_id, job_class_str, output):
+        """Extracts the results of a MR job and registers its completion.
+
+        Args:
+            job_id: str. The ID of the job to run.
+            job_class_str: str. Should uniquely identify each type of job.
+            output: str. The output produced by the job.
+        """
         job_class = mapreduce_util.for_name(job_class_str)
 
         try:
@@ -445,23 +610,41 @@ class GoogleCloudStorageConsistentJsonOutputWriter(
         output_writers.GoogleCloudStorageConsistentOutputWriter):
 
     def write(self, data):
+        """Writes that data serialized in JSON format.
+
+        Args:
+            data: *. Data to be serialized in JSON format.
+        """
         super(GoogleCloudStorageConsistentJsonOutputWriter, self).write(
             '%s\n' % json.dumps(data))
 
 
 class BaseMapReduceJobManager(BaseJobManager):
-    # The output for this job is a list of individual results. Each item in
-    # the list will be of whatever type is yielded from the 'reduce' method.
-    #
-    # The 'metadata' field in the BaseJob representing a MapReduceJob
-    # is a dict with one key, _OUTPUT_KEY_ROOT_PIPELINE_ID. The corresponding
-    # value is a string representing the ID of the MapReduceJobPipeline
-    # as known to the mapreduce/lib/pipeline internals. This is used
-    # to generate URLs pointing at the pipeline support UI.
+    """The output for this job is a list of individual results. Each item in the
+    list will be of whatever type is yielded from the 'reduce' method.
+
+    The 'metadata' field in the BaseJob representing a MapReduceJob is a dict
+    with one key, _OUTPUT_KEY_ROOT_PIPELINE_ID. The corresponding value is a
+    string representing the ID of the MapReduceJobPipeline as known to the
+    mapreduce/lib/pipeline internals. This is used to generate URLs pointing at
+    the pipeline support UI.
+    """
     _OUTPUT_KEY_ROOT_PIPELINE_ID = 'root_pipeline_id'
 
     @staticmethod
     def get_mapper_param(param_name):
+        """Returns current value of given param_name for this job.
+
+        Args:
+            param_name: str. One of the configurable parameters of this
+                particular mapreduce job.
+
+        Returns:
+            *. The current value of the parameter.
+
+        Raises:
+            Exception: The parameter is not associated to this job type.
+        """
         mapper_params = context.get().mapreduce_spec.mapper.params
         if param_name not in mapper_params:
             raise Exception(
@@ -470,25 +653,25 @@ def get_mapper_param(param_name):
 
     @classmethod
     def entity_classes_to_map_over(cls):
-        """Return a list of reference to the datastore classes to map over."""
+        """Return a list of datastore class references to map over."""
         raise NotImplementedError(
             'Classes derived from BaseMapReduceJobManager must implement '
             'entity_classes_to_map_over()')
 
     @staticmethod
     def map(item):
-        """Implements the map function.  Must be declared @staticmethod.
+        """Implements the map function. Must be declared @staticmethod.
+
+        This function may yield as many times as appropriate (including zero)
+        to return key/value 2-tuples. For example, to get a count of all
+        explorations, one might yield (exploration.id, 1).
+
+        WARNING: The OutputWriter converts mapper output keys to type str. So,
+        if you have keys that are of type unicode, you must yield
+        "key.encode('utf-8')", rather than "key".
 
         Args:
-          item: The parameter passed to this function is a single element of
-          the type given by entity_class(). This function may yield as many
-          times as appropriate (including zero) to return key/value 2-tuples.
-          For example, to get a count of all explorations, one might yield
-          (exploration.id, 1).
-
-          WARNING: The OutputWriter converts mapper output keys to type str.
-          So, if you have keys that are of type unicode, you must yield
-          "key.encode('utf-8')", rather than "key".
+            item: *. A single element of the type given by entity_class().
         """
         raise NotImplementedError(
             'Classes derived from BaseMapReduceJobManager must implement map '
@@ -496,7 +679,7 @@ def map(item):
 
     @staticmethod
     def reduce(key, values):
-        """Implements the reduce function.  Must be declared @staticmethod.
+        """Implements the reduce function. Must be declared @staticmethod.
 
         This function should yield a JSON string. All emitted outputs from all
         reducers will be collected in an array and set into the output value
@@ -504,13 +687,16 @@ def reduce(key, values):
         persist it out into the datastore instead and return a reference (and
         dereference it later to load content as needed).
 
+        This code can assume that it is the only process handling values for the
+        given key.
+
+        TODO(brianrodri): Verify whether it can also assume that it will be
+        called exactly once for each key with all of the output.
+
         Args:
-          key: A key value as emitted from the map() function, above.
-          values: A list of all values from all mappers that were tagged with
-          the given key. This code can assume that it is the only process
-          handling values for this key. (It can probably also assume that
-          it will be called exactly once for each key with all of the output,
-          but this needs to be verified.)
+            key: *. A key value as emitted from the map() function, above.
+            values: list(*). A list of all values from all mappers that were
+                tagged with the given key.
         """
         raise NotImplementedError(
             'Classes derived from BaseMapReduceJobManager must implement '
@@ -518,10 +704,21 @@ def reduce(key, values):
 
     @classmethod
     def _real_enqueue(cls, job_id, additional_job_params):
+        """Configures, creates, and queues the pipeline for the given job and
+        params.
+
+        Args:
+            job_id: str. The ID of the job to enqueue.
+            additional_job_params: dict(str : *). Additional params to pass into
+                the job's _run() method.
+
+        Raises:
+            Exception: Passed a value to a parameter in the mapper which has
+                already been given a value.
+        """
         entity_class_types = cls.entity_classes_to_map_over()
         entity_class_names = [
-            '%s.%s' % (
-                entity_class_type.__module__, entity_class_type.__name__)
+            '%s.%s' % (entity_class_type.__module__, entity_class_type.__name__)
             for entity_class_type in entity_class_types]
 
         kwargs = {
@@ -577,6 +774,13 @@ def _entity_created_before_job_queued(entity):
         Mapper methods may want to use this as a precomputation check,
         especially if the datastore classes being iterated over are append-only
         event logs.
+
+        Args:
+            entity: BaseModel. An entity this job type is responsible for
+                handling.
+
+        Returns:
+            bool. Whether the entity was queued before the job was created.
         """
         created_on_msec = utils.get_time_in_millisecs(entity.created_on)
         job_queued_msec = float(context.get().mapreduce_spec.mapper.params[
@@ -597,16 +801,44 @@ def __iter__(self):
 
     @classmethod
     def from_json(cls, input_shard_state):
+        """Creates an instance of the InputReader for the given input shard
+        state.
+
+        Args:
+            input_shard_state: dict(str : *). The InputReader state as a
+                dict-like object.
+
+        Returns:
+            *. An instance of the InputReader configured using the input shard
+                state.
+        """
         return cls(input_readers.DatastoreInputReader.from_json(
             input_shard_state[cls._READER_LIST_PARAM]))
 
     def to_json(self):
+        """Returns an input shard state for the remaining inputs.
+
+        Returns:
+            dict(str : *). A json-izable version of the remaining InputReader.
+        """
         return {
             self._READER_LIST_PARAM: self._reader_list.to_json()
         }
 
     @classmethod
     def split_input(cls, mapper_spec):
+        """Returns a list of input readers.
+
+        This method creates a list of input readers, each for one shard. It
+        attempts to split inputs among readers evenly.
+
+        Args:
+            mapper_spec: model.MapperSpec. Specifies the inputs and additional
+                parameters to define the behavior of input readers.
+
+        Returns:
+            list(InputReaders). None or [] when no input data can be found.
+        """
         params = mapper_spec.params
         entity_kinds = params.get(cls._ENTITY_KINDS_PARAM)
 
@@ -625,7 +857,22 @@ def split_input(cls, mapper_spec):
         return inputs
 
     @classmethod
-    def validate(cls, mapper_spec):
+    def validate(cls, unused_mapper_spec):
+        """Validates mapper spec and all mapper parameters.
+
+        Input reader parameters are expected to be passed as "input_reader"
+        subdictionary in mapper_spec.params.
+
+        Pre 1.6.4 API mixes input reader parameters with all other parameters.
+        Thus to be compatible, input reader check mapper_spec.params as well and
+        issue a warning if "input_reader" subdicationary is not present.
+
+        Args:
+            mapper_spec: model.MapperSpec. The MapperSpec for this InputReader.
+
+        Raises:
+            BadReaderParamsError: Required parameters are missing or invalid.
+        """
         return True  # TODO
 
 
@@ -642,6 +889,9 @@ def _get_continuous_computation_class(cls):
 
     @staticmethod
     def _get_job_queued_msec():
+        """Returns the time when the job got queued, in milliseconds past the
+        Epoch.
+        """
         return float(context.get().mapreduce_spec.mapper.params[
             MAPPER_PARAM_KEY_QUEUED_TIME_MSECS])
 
@@ -653,6 +903,14 @@ def _entity_created_before_job_queued(entity):
         Mapper methods may want to use this as a precomputation check,
         especially if the datastore classes being iterated over are append-only
         event logs.
+
+        Args:
+            entity: BaseModel. An entity this job type is responsible for
+                handling.
+
+        Returns:
+            bool. Whether the entity was created before the given MR job was
+                queued.
         """
         created_on_msec = utils.get_time_in_millisecs(entity.created_on)
         job_queued_msec = float(context.get().mapreduce_spec.mapper.params[
@@ -685,15 +943,22 @@ class BaseRealtimeDatastoreClassForContinuousComputations(
     the 0 or 1 indicates the realtime layer that the entity is in.
 
     NOTE TO DEVELOPERS: Ensure that you wrap the id with get_realtime_id()
-    when doing creations, gets, puts and queries, in order to ensure that the
+    when doing creations, gets, puts, and queries. This ensures that the
     relevant layer prefix gets appended.
     """
     realtime_layer = ndb.IntegerProperty(required=True, choices=[0, 1])
 
     @classmethod
     def get_realtime_id(cls, layer_index, raw_entity_id):
-        """Returns an ID used to identify the element with the given entity id
-        in the currently active realtime datastore layer.
+        """Returns a valid id for the given (realtime layer, entity) pair.
+
+        Args:
+            layer_index: int. The realtime layer id, one of: 0, 1.
+            raw_entity_id: int. The id of an entity that is to be stored in the
+                given realtime layer.
+
+        Returns:
+            str. Uniquely identifies the (realtime layer, entity) pair.
         """
         return '%s:%s' % (layer_index, raw_entity_id)
 
@@ -701,6 +966,12 @@ def get_realtime_id(cls, layer_index, raw_entity_id):
     def delete_layer(cls, layer_index, latest_created_on_datetime):
         """Deletes all entities in the given layer which were created before
         the given datetime.
+
+        Args:
+            layer_index: int. The realtime layer id, should be either 0 or 1.
+            latest_created_on_datetime: datetime.datetime. All remaining
+                entities after the deletion will have been created on or after
+                this time.
         """
         query = cls.query().filter(cls.realtime_layer == layer_index).filter(
             cls.created_on < latest_created_on_datetime)
@@ -708,10 +979,36 @@ def delete_layer(cls, layer_index, latest_created_on_datetime):
 
     @classmethod
     def _is_valid_realtime_id(cls, realtime_id):
+        """Returns whether the realtime_id represents a valid (realtime layer,
+        entity) pair.
+
+        Args:
+            realtime_id: str. The id to query.
+
+        Returns:
+            bool. Whether the realtime_id represents a valid (realtime layer,
+                entity) pair.
+        """
         return realtime_id.startswith('0:') or realtime_id.startswith('1:')
 
     @classmethod
     def get(cls, entity_id, strict=True):
+        """Gets an entity by its id.
+
+        Args:
+            entity_id: str. Unique identifier for an entity.
+            strict: bool. Whether to fail noisily if no entity with the given
+                id exists in the datastore. Default is True.
+
+        Returns:
+            * or None. The entity instance that corresponds to the given id, or
+                None if strict == False and no undeleted entity with the given
+                id exists in the datastore.
+
+        Raises:
+            base_models.BaseModel.EntityNotFoundError: strict == True and no
+                undeleted entity with the given id exists in the datastore.
+        """
         if not cls._is_valid_realtime_id(entity_id):
             raise ValueError('Invalid realtime id: %s' % entity_id)
 
@@ -720,6 +1017,11 @@ def get(cls, entity_id, strict=True):
         ).get(entity_id, strict=strict)
 
     def put(self):
+        """Stores the current realtime layer entity into the database.
+
+        Raises:
+            Exception: The current instance has an invalid realtime layer id.
+        """
         if (self.realtime_layer is None or
                 str(self.realtime_layer) != self.id[0]):
             raise Exception(
@@ -732,6 +1034,7 @@ def put(self):
 
 class BaseContinuousComputationManager(object):
     """This class represents a manager for a continuously-running computation.
+
     Such computations consist of two parts: a batch job to compute summary
     views, and a realtime layer to augment these batch views with additional
     data that has come in since the last batch job results were computed. The
@@ -792,6 +1095,7 @@ def _get_realtime_datastore_class(cls):
     @classmethod
     def _get_batch_job_manager_class(cls):
         """Returns the manager class for the continuously-running batch job.
+
         See jobs_test.py for an example of how to do this.
         """
         raise NotImplementedError(
@@ -806,7 +1110,8 @@ def _handle_incoming_event(
 
         This method should be implemented by subclasses. The args are the
         same as those sent to the event handler corresponding to the event
-        type. Note that there may be more than one event type.
+        type, so check that for documentation. Note that there may be more than
+        one event type.
 
         IMPORTANT: This method only gets called as part of the dequeue process
         from a deferred task queue. Developers should expect a delay to occur
@@ -820,6 +1125,16 @@ def _handle_incoming_event(
         exception has type taskqueue_services.PermanentTaskFailure. Developers
         should therefore ensure that _handle_incoming_event() is robust to
         multiple calls for the same incoming event.
+
+        Args:
+            active_realtime_layer: int. The currently active realtime
+                datastore layer.
+            event_type: str. The event triggered by a student. For example, when
+                a student starts an exploration, event of type `start` is
+                triggered. If he/she completes an exploration, event of type
+                `complete` is triggered.
+            *args: list(*). Forwarded to the _handle_event() method.
+            **kwargs: dict(* : *). Forwarded to the _handle_event() method.
         """
         raise NotImplementedError(
             'Subclasses of BaseContinuousComputationManager must implement '
@@ -829,6 +1144,14 @@ def _handle_incoming_event(
 
     @classmethod
     def _get_active_realtime_index(cls):
+        """Returns the currently active realtime layer index for this class.
+
+        Additionally, takes care to register the relevant
+        ContinuousComputationModel, if it does not already exist.
+
+        Returns:
+            str. The active realtime layer index of this class.
+        """
         def _get_active_realtime_index_transactional():
             cc_model = job_models.ContinuousComputationModel.get(
                 cls.__name__, strict=False)
@@ -846,6 +1169,13 @@ def _get_active_realtime_index_transactional():
     def get_active_realtime_layer_id(cls, entity_id):
         """Returns an ID used to identify the element with the given entity id
         in the currently active realtime datastore layer.
+
+        Args:
+            entity_id: str. Unique identifier for an entity.
+
+        Returns:
+            str. Unique identifier for the given entity for storage in active
+                realtime layer.
         """
         return cls._get_realtime_datastore_class().get_realtime_id(
             cls._get_active_realtime_index(), entity_id)
@@ -854,6 +1184,13 @@ def get_active_realtime_layer_id(cls, entity_id):
     def get_multi_active_realtime_layer_ids(cls, entity_ids):
         """Returns a list of IDs of elements in the currently active realtime
         datastore layer corresponding to the given entity ids.
+
+        Args:
+            entity_ids: str. Collection of unique identifiers for entities.
+
+        Returns:
+            list(str). Unique identifiers for each given entity for storage in
+                the currently active realtime layer.
         """
         realtime_datastore_class = cls._get_realtime_datastore_class()
         active_realtime_index = cls._get_active_realtime_index()
@@ -864,6 +1201,9 @@ def get_multi_active_realtime_layer_ids(cls, entity_ids):
 
     @classmethod
     def _switch_active_realtime_class(cls):
+        """Switches the currently-active realtime layer for this continuous
+        computation.
+        """
         def _switch_active_realtime_class_transactional():
             cc_model = job_models.ContinuousComputationModel.get(
                 cls.__name__)
@@ -875,10 +1215,13 @@ def _switch_active_realtime_class_transactional():
             _switch_active_realtime_class_transactional)
 
     @classmethod
-    def _clear_inactive_realtime_layer(
-            cls, latest_created_on_datetime):
-        """Deletes all entries in the given realtime datastore class whose
-        created_on date is before latest_timestamp.
+    def _clear_inactive_realtime_layer(cls, latest_created_on_datetime):
+        """Deletes all entries in the inactive realtime datastore class whose
+        created_on date is before latest_created_on_datetime.
+
+        Args:
+            latest_created_on_datetime: datetime.datetime. Entities created
+                before this datetime will be removed.
         """
         inactive_realtime_index = 1 - cls._get_active_realtime_index()
         cls._get_realtime_datastore_class().delete_layer(
@@ -909,7 +1252,8 @@ def _register_end_of_batch_job_and_return_status(cls):
         """
         def _register_end_of_batch_job_transactional():
             """Transactionally change the computation's status when a batch job
-            ends."""
+            ends.
+            """
             cc_model = job_models.ContinuousComputationModel.get(cls.__name__)
             if (cc_model.status_code ==
                     job_models.CONTINUOUS_COMPUTATION_STATUS_CODE_STOPPING):
@@ -932,7 +1276,8 @@ def get_status_code(cls):
     def start_computation(cls):
         """(Re)starts the continuous computation corresponding to this class.
 
-        Raises an Exception if the computation is already running.
+        Raises:
+            Exception: The computation wasn't idle before trying to start.
         """
         def _start_computation_transactional():
             """Transactional implementation for marking a continuous
@@ -963,10 +1308,13 @@ def _start_computation_transactional():
         cls._kickoff_batch_job()
 
     @classmethod
-    def stop_computation(cls, user_id, unused_test_mode=False):
+    def stop_computation(cls, user_id):
         """Cancels the currently-running batch job.
 
         No further batch runs will be kicked off.
+
+        Args:
+            user_id: str. The id of the user stopping the job.
         """
         # This is not an ancestor query, so it must be run outside a
         # transaction.
@@ -1002,11 +1350,19 @@ def _stop_computation_transactional():
 
     @classmethod
     def on_incoming_event(cls, event_type, *args, **kwargs):
-        """Handle an incoming event by recording it in both realtime datastore
+        """Handles an incoming event by recording it in both realtime datastore
         layers.
 
         The *args and **kwargs match those passed to the _handle_event() method
         of the corresponding EventHandler subclass.
+
+        Args:
+            event_type: str. The event triggered by a student. For example, when
+                a student starts an exploration, event of type `start` is
+                triggered. If he/she completes an exploration, event of type
+                `complete` is triggered.
+            *args: Forwarded to _handle_event() method.
+            *kwargs: Forwarded to _handle_event() method.
         """
         realtime_layers = [0, 1]
         for layer in realtime_layers:
@@ -1014,12 +1370,15 @@ def on_incoming_event(cls, event_type, *args, **kwargs):
 
     @classmethod
     def _process_job_completion_and_return_status(cls):
-        """Delete all data in the currently-active realtime_datastore class,
-        switch the active class, and return the status.
+        """Deletes all data in the currently-active realtime_datastore class,
+        then switches the active class.
 
         This seam was created so that tests would be able to override
         on_batch_job_completion() to avoid kicking off the next job
         immediately.
+
+        Returns:
+            str. The final status of the job.
         """
         cls._switch_active_realtime_class()
         cls._clear_inactive_realtime_layer(datetime.datetime.utcnow())
@@ -1041,16 +1400,17 @@ def _kickoff_batch_job_after_previous_one_ends(cls):
 
     @classmethod
     def on_batch_job_completion(cls):
-        """Called when a batch job completes."""
+        """Marks the existing job as complete and kicks off a new batch job."""
         job_status = cls._process_job_completion_and_return_status()
         if job_status == job_models.CONTINUOUS_COMPUTATION_STATUS_CODE_RUNNING:
             cls._kickoff_batch_job_after_previous_one_ends()
 
     @classmethod
     def on_batch_job_canceled(cls):
+        """Marks the batch job as cancelled and verifies that the continuous
+        computation is now idle.
+        """
         logging.info('Job %s canceled.' % cls.__name__)
-        # The job should already be stopping, and should therefore be marked
-        # idle.
         job_status = cls._register_end_of_batch_job_and_return_status()
         if job_status != job_models.CONTINUOUS_COMPUTATION_STATUS_CODE_IDLE:
             logging.error(
@@ -1059,6 +1419,7 @@ def on_batch_job_canceled(cls):
 
     @classmethod
     def on_batch_job_failure(cls):
+        """Gives up on the existing batch job and kicks off a new one."""
         # TODO(sll): Alert the site admin via email.
         logging.error('Job %s failed.' % cls.__name__)
         job_status = cls._register_end_of_batch_job_and_return_status()
@@ -1069,20 +1430,26 @@ def on_batch_job_failure(cls):
 def _get_job_dict_from_job_model(model):
     """Converts an ndb.Model representing a job to a dict.
 
-    The dict contains the following keys:
-    - 'id': the job id
-    - 'time_started_msec': when the job was started, in milliseconds since the
-          epoch
-    - 'time_finished_msec': when the job was finished, in milliseconds since
-          the epoch
-    - 'status_code': the current status of the job
-    - 'job_type': the type of this job
-    - 'is_cancelable': whether the job can be canceled
-    - 'error': any errors pertaining to this job
-    - 'human_readable_time_started': a human-readable string representing the
-          time the job started, or None if time_started_msec is None.
-    - 'human_readable_time_finished': a human-readable string representing the
-          time the job finished, or None if time_finished_msec is None.
+    Args:
+        model: ndb.Model. The model to extract job info from.
+
+    Returns:
+        dict. The dict contains the following keys:
+            id: str. The ID of the job.
+            time_started_msec: float. When the job was started, in milliseconds
+                since the epoch.
+            time_finished_msec: float. When the job was finished, in
+                milliseconds since the epoch.
+            status_code: str. The current status of the job.
+            job_type: str. The type of this job.
+            is_cancelable: bool. Whether the job can be canceled.
+            error: str. Any errors pertaining to this job.
+            human_readable_time_started: str or None. A human-readable string
+                representing the time the job started, or None if
+                time_started_msec is None.
+            human_readable_time_finished: str or None. A human-readable string
+                representing the time the job finished, or None if
+                time_finished_msec is None.
     """
     return {
         'id': model.id,
@@ -1110,7 +1477,25 @@ def get_data_for_recent_jobs(recency_msec=DEFAULT_RECENCY_MSEC):
     returned.
 
     Args:
-        recency_msec: the threshold for a recent job, in milliseconds.
+        recency_msec: int. The threshold for a recent job, in milliseconds.
+
+    Returns:
+        list(dict). Each dict contains the following keys:
+            id: str. The ID of the job.
+            time_started_msec: float. When the job was started, in milliseconds
+                since the epoch.
+            time_finished_msec: float. When the job was finished, in
+                milliseconds since the epoch.
+            status_code: str. The current status of the job.
+            job_type: str. The type of this job.
+            is_cancelable: bool. Whether the job can be canceled.
+            error: str. Any errors pertaining to this job.
+            human_readable_time_started: str or None. A human-readable string
+                representing the time the job started, or None if
+                time_started_msec is None.
+            human_readable_time_finished: str or None. A human-readable string
+                representing the time the job finished, or None if
+                time_finished_msec is None.
     """
     recent_job_models = job_models.JobModel.get_recent_jobs(
         NUM_JOBS_IN_DASHBOARD_LIMIT, recency_msec)
@@ -1118,7 +1503,28 @@ def get_data_for_recent_jobs(recency_msec=DEFAULT_RECENCY_MSEC):
 
 
 def get_data_for_unfinished_jobs():
-    """Get a list containing data about all unfinished jobs."""
+    """Returns a list of dicts containing data about all unfinished jobs.
+
+    Returns:
+        list(dict). Each dict represents a continuous computation and contains
+            the following keys:
+                computation_type: str. The type of the computation.
+                status_code: str. The current status of the computation.
+                last_started_msec: float or None. When a batch job for the
+                    computation was last started, in milliseconds since the
+                    epoch.
+                last_finished_msec: float or None. When a batch job for the
+                    computation last finished, in milliseconds since the epoch.
+                last_stopped_msec: float or None. When a batch job for the
+                    computation was last stopped, in milliseconds since the
+                    epoch.
+                active_realtime_layer_index: int or None. The index of the
+                    active realtime layer.
+                is_startable: bool. Whether an admin should be allowed to start
+                    this computation.
+                is_stoppable: bool. Whether an admin should be allowed to stop
+                    this computation.
+    """
     unfinished_job_models = job_models.JobModel.get_all_unfinished_jobs(
         NUM_JOBS_IN_DASHBOARD_LIMIT)
     return [_get_job_dict_from_job_model(model)
@@ -1126,7 +1532,14 @@ def get_data_for_unfinished_jobs():
 
 
 def get_job_output(job_id):
-    """Returns the output of a job."""
+    """Returns the output of a job.
+
+    Args:
+        job_id: str. The ID of the job to query.
+
+    Returns:
+        str. The result of the job.
+    """
     return job_models.JobModel.get_by_id(job_id).output
 
 
@@ -1134,24 +1547,28 @@ def get_continuous_computations_info(cc_classes):
     """Returns data about the given computations.
 
     Args:
-      cc_classes: a list of subclasses of BaseContinuousComputationManager.
+        cc_classes: list(BaseContinuousComputationManager). List of continuous
+            computation subclasses.
 
     Returns:
-      A list of dicts, each representing a continuous computation. Each dict
-      has the following keys:
-      - 'computation_type': the type of the computation
-      - 'status_code': the current status of the computation
-      - 'last_started_msec': when a batch job for the computation was last
-            started, in milliseconds since the epoch
-      - 'last_finished_msec': when a batch job for the computation last
-            finished, in milliseconds since the epoch
-      - 'last_stopped_msec': when a batch job for the computation was last
-            stopped, in milliseconds since the epoch
-      - 'active_realtime_layer_index': the index of the active realtime layer
-      - 'is_startable': whether an admin should be allowed to start this
-            computation
-      - 'is_stoppable': whether an admin should be allowed to stop this
-            computation
+        list(dict). Each dict represents a continuous computation and contains
+            the following keys:
+                computation_type: str. The type of the computation.
+                status_code: str. The current status of the computation.
+                last_started_msec: float or None. When a batch job for the
+                    computation was last started, in milliseconds since the
+                    epoch.
+                last_finished_msec: float or None. When a batch job for the
+                    computation last finished, in milliseconds since the epoch.
+                last_stopped_msec: float or None. When a batch job for the
+                    computation was last stopped, in milliseconds since the
+                    epoch.
+                active_realtime_layer_index: int or None. The index of the
+                    active realtime layer.
+                is_startable: bool. Whether an admin should be allowed to start
+                    this computation.
+                is_stoppable: bool. Whether an admin should be allowed to stop
+                    this computation.
     """
     cc_models = job_models.ContinuousComputationModel.get_multi(
         [cc_class.__name__ for cc_class in cc_classes])
@@ -1193,7 +1610,12 @@ def get_continuous_computations_info(cc_classes):
 
 def get_stuck_jobs(recency_msecs):
     """Returns a list of jobs which were last updated at most recency_msecs
-    milliseconds ago and have experienced more than one retry."""
+    milliseconds ago and have experienced more than one retry.
+
+    Returns:
+        list(job_models.JobModel). Jobs which have retried at least once and
+            haven't finished yet.
+    """
     threshold_time = (
         datetime.datetime.utcnow() -
         datetime.timedelta(0, 0, 0, recency_msecs))
@@ -1216,6 +1638,7 @@ class JobCleanupManager(BaseMapReduceJobManager):
 
     @classmethod
     def entity_classes_to_map_over(cls):
+        """The entity types this job will handle."""
         return [
             mapreduce_model.MapreduceState,
             mapreduce_model.ShardState
@@ -1223,6 +1646,16 @@ def entity_classes_to_map_over(cls):
 
     @staticmethod
     def map(item):
+        """Implements the map function which will clean up jobs that have not
+        finished.
+
+        Args:
+            item: mapreduce_model.MapreduceState or mapreduce_model.ShardState.
+                A shard or job which may still be running.
+        Yields:
+            tuple(str, int). Describes the action taken for the item, and the
+                number of items this action was applied to.
+        """
         max_start_time_msec = JobCleanupManager.get_mapper_param(
             MAPPER_PARAM_MAX_START_TIME_MSEC)
 
@@ -1246,6 +1679,16 @@ def map(item):
 
     @staticmethod
     def reduce(key, stringified_values):
+        """Implements the reduce function which logs the results of the mapping
+        function.
+
+        Args:
+            key: str. Describes the action taken by a map call. One of:
+                'mr_state_deleted', 'mr_state_remaining', 'shard_state_deleted',
+                'shard_state_remaining'.
+            stringified_values: list(str). A list where each element is a
+                stringified number, counting the mapped items sharing the key.
+        """
         values = [ast.literal_eval(v) for v in stringified_values]
         if key.endswith('_deleted'):
             logging.warning(


Test patch: