Repository: oppia/oppia

Issue:
Start using transaction decorator instead of  `run_in_transaction`
Replace all usages of `run_in_transaction` with the new `run_in_transaction_wrapper`.

### List of files
- [x] email_manager.py @Nik-09
- [x] feedback_jobs_continuous.py @Nik-09
- [x] feedback_services.py @lkbhitesh07
- [x] question_services.py @lkbhitesh07
- [x] rating_services.py @vojtechjelinek (#11741)
- [x] stats_services.py @lkbhitesh07
- [x] suggestion_jobs_one_off.py @lkbhitesh07
- [x] suggestion_services.py @lkbhitesh07
- [x] user_services.py @lkbhitesh07
- [x] wipeout_services.py @OBITORASU 
- [x] jobs.py @lkbhitesh07
- [x] base_model/gae_models.py @OBITORASU 
- [x] statistics/gae_models.py @lkbhitesh07
- [x] user_jobs_continuous.py @lkbhitesh07

### Steps to refactor
1. Claim one of the files listed above by pinging @vojtechjelinek.
2. Look at the occurrences of `run_in_transaction` in the claimed file.
3. Instead of doing for example `transaction_services.run_in_transaction(_update_user_rating)` on top of the definition of `_update_user_rating` the `@transaction_services.run_in_transaction_wrapper` decorator should be added. In the callsite, the `transaction_services.run_in_transaction(_update_user_rating)` should be replaced with `_update_user_rating_transactional()`.
4. See #11741 for reference

After all occurences are replaced remame the `run_in_transaction_wrapper` to `run_in_transaction`.
Start using transaction decorator instead of  `run_in_transaction`
Replace all usages of `run_in_transaction` with the new `run_in_transaction_wrapper`.

### List of files
- [x] email_manager.py @Nik-09
- [x] feedback_jobs_continuous.py @Nik-09
- [x] feedback_services.py @lkbhitesh07
- [x] question_services.py @lkbhitesh07
- [x] rating_services.py @vojtechjelinek (#11741)
- [x] stats_services.py @lkbhitesh07
- [x] suggestion_jobs_one_off.py @lkbhitesh07
- [x] suggestion_services.py @lkbhitesh07
- [x] user_services.py @lkbhitesh07
- [x] wipeout_services.py @OBITORASU 
- [x] jobs.py @lkbhitesh07
- [x] base_model/gae_models.py @OBITORASU 
- [x] statistics/gae_models.py @lkbhitesh07
- [x] user_jobs_continuous.py @lkbhitesh07

### Steps to refactor
1. Claim one of the files listed above by pinging @vojtechjelinek.
2. Look at the occurrences of `run_in_transaction` in the claimed file.
3. Instead of doing for example `transaction_services.run_in_transaction(_update_user_rating)` on top of the definition of `_update_user_rating` the `@transaction_services.run_in_transaction_wrapper` decorator should be added. In the callsite, the `transaction_services.run_in_transaction(_update_user_rating)` should be replaced with `_update_user_rating_transactional()`.
4. See #11741 for reference

After all occurences are replaced remame the `run_in_transaction_wrapper` to `run_in_transaction`.


Discussion:
#11404 for context.
@vojtechjelinek can you please assign me these two files.
 email_manager.py
 feedback_jobs_continuous.py
 Thanks.
Hello @vojtechjelinek , 
Will you please assign `user_jobs_continuous.py` file to me ?
Thanks
@Nik-09 Done.

@lkbhitesh07 Done, can you please also follow these instructions https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia in order to contribute to Oppia.
Hello @vojtechjelinek 
Sorry for not updating you, I was facing some troubles in my environment but now I am good to go. Also, I want to ask one question that do we have to make PR for each part of this issue, or we can simply do with 2 to 3 parts?
Thanks
> Hello @vojtechjelinek
> Sorry for not updating you, I was facing some troubles in my environment but now I am good to go. Also, I want to ask one question that do we have to make PR for each part of this issue, or we can simply do with 2 to 3 parts?
> Thanks

@lkbhitesh07 Up to you, the changes are quite small, so they can be done in one PR.
>  @lkbhitesh07 Up to you, the changes are quite small, so they can be done in one PR.

Ok Thanks @vojtechjelinek ,
Will you please assign these files to me.
`jobs.py` 
`base_model/gae_models.py`
`statistics/gae_models.py`


Hi @vojtechjelinek 
can I please go ahead with the rest of the files, I'm halfway done?
Thanks
@lkbhitesh07 Let's leave that to other new contributors, you can take up other issues. Btw did you fill the [Oppia contributor survey](https://goo.gl/forms/otv30JV3Ihv0dT3C3)?
> @lkbhitesh07 Let's leave that to other new contributors, you can take up other issues. Btw did you fill the [Oppia contributor survey](https://goo.gl/forms/otv30JV3Ihv0dT3C3)?

@vojtechjelinek Yes I already filled the form when I contributed few years back, also I'm done with all the files here except `statistics/gae_models.py` and `wipeout_services.py` . Can I please make a PR?
Sorry for that, I'll keep it in mind from next time.
@lkbhitesh07 Okay I found you in our system. Sure make the PR for these or include them in the PR that you already made.
@vojtechjelinek Thank you so much.
Hello, @vojtechjelinek I'm done with all the files except `statistics/gae_models.py` and `wipeout_services.py`. As you suggested and discussed above will be leaving these 2 files for new contributors.
Also please review these PRs #11782 #11783 
Thanks for all your support.
Hello @vojtechjelinek 
I figured out that in my previous comment I accidentally mentioned `statistics/gae_models.py` instead of `base_model/gae_models.py`. Will you please correct it. I'm done with `statistics/gae_models.py` and waiting for your approval.
Thanks
@vojtechjelinek can I take up the two remaining files ```base_model/gae_models.py``` and ``` wipeout_services.py```? I was suggested to work on this issue as a part of my onboarding.
@OBITORASU Sure, I have assigned those to you.
@vojtechjelinek I had a doubt regarding ```base_model/gae_models.py```
So to elaborate, we have a classmethod namely:
``` 
@classmethod
    def delete_multi(cls, entities):
        """Deletes the given datastore_services.Model instances.

        Args:
            entities: list(datastore_services.Model). The list of model
                instances to be deleted.
        """
        datastore_services.delete_multi([entity.key for entity in entities])
```
Now datastore_services is defined as:
```
datastore_services = models.Registry.import_datastore_services()
```
So till here I have a clear idea of what is going on. Now the issue is when searching for ```transaction_services.run_in_transaction``` I came across this particular line of code:
```
transaction_services.run_in_transaction(
                    datastore_services.delete_multi,
                    all_models_keys[
                        i:i + feconf.MAX_NUMBER_OF_OPS_IN_TRANSACTION])
```
Now my question is why have a classmethod delete_multi if we are not using BaseModel.delete_multi anywhere in code of the particular file. As there has been no instance of BaseModel.delete_multi when I try to find it.

Images for reference:
![Screenshot_20210129_113536](https://user-images.githubusercontent.com/65222459/106238851-99e37780-6227-11eb-9dfe-96b1ce788e30.png)

![Screenshot_20210129_113552](https://user-images.githubusercontent.com/65222459/106238880-a962c080-6227-11eb-9f04-067932d5b068.png)

![Screenshot_20210129_113813](https://user-images.githubusercontent.com/65222459/106238901-b4b5ec00-6227-11eb-9a1a-2648b4ffee5e.png)

![image](https://user-images.githubusercontent.com/65222459/106238996-dfa04000-6227-11eb-905a-6f200387a8e5.png)




Also another question regarding the ```wipeout_services.py```
Just to confirm, is it ```wipeout_services.py``` or ```wipeout_service.py``` ?
Apparently I can't fuzzy find ```wipeout_services.py``` the closest file to it is ```wipeout_service.py``` which I can find.
@OBITORASU The `BaseModel.delete_multi` can be used in other files, also since some models inherit from `BaseModel` (like `UserSettingsModel`), the `delete_multi` can be used in these classes too.
Thanks for the explanation :)
@Nik-09 How is your work going? Do you need help with anything?
> @Nik-09 How is your work going? Do you need help with anything?

I have created Pr please ask @aks681 to review [pr link](https://github.com/oppia/oppia/pull/11781)
Thanks @vojtechjelinek and @aks681 .

Hi @vojtechjelinek , Is there any task available in this issue?
@17sushmita No, it isn't.
#11404 for context.
@vojtechjelinek can you please assign me these two files.
 email_manager.py
 feedback_jobs_continuous.py
 Thanks.
Hello @vojtechjelinek , 
Will you please assign `user_jobs_continuous.py` file to me ?
Thanks
@Nik-09 Done.

@lkbhitesh07 Done, can you please also follow these instructions https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia in order to contribute to Oppia.
Hello @vojtechjelinek 
Sorry for not updating you, I was facing some troubles in my environment but now I am good to go. Also, I want to ask one question that do we have to make PR for each part of this issue, or we can simply do with 2 to 3 parts?
Thanks
> Hello @vojtechjelinek
> Sorry for not updating you, I was facing some troubles in my environment but now I am good to go. Also, I want to ask one question that do we have to make PR for each part of this issue, or we can simply do with 2 to 3 parts?
> Thanks

@lkbhitesh07 Up to you, the changes are quite small, so they can be done in one PR.
>  @lkbhitesh07 Up to you, the changes are quite small, so they can be done in one PR.

Ok Thanks @vojtechjelinek ,
Will you please assign these files to me.
`jobs.py` 
`base_model/gae_models.py`
`statistics/gae_models.py`


Hi @vojtechjelinek 
can I please go ahead with the rest of the files, I'm halfway done?
Thanks
@lkbhitesh07 Let's leave that to other new contributors, you can take up other issues. Btw did you fill the [Oppia contributor survey](https://goo.gl/forms/otv30JV3Ihv0dT3C3)?
> @lkbhitesh07 Let's leave that to other new contributors, you can take up other issues. Btw did you fill the [Oppia contributor survey](https://goo.gl/forms/otv30JV3Ihv0dT3C3)?

@vojtechjelinek Yes I already filled the form when I contributed few years back, also I'm done with all the files here except `statistics/gae_models.py` and `wipeout_services.py` . Can I please make a PR?
Sorry for that, I'll keep it in mind from next time.
@lkbhitesh07 Okay I found you in our system. Sure make the PR for these or include them in the PR that you already made.
@vojtechjelinek Thank you so much.
Hello, @vojtechjelinek I'm done with all the files except `statistics/gae_models.py` and `wipeout_services.py`. As you suggested and discussed above will be leaving these 2 files for new contributors.
Also please review these PRs #11782 #11783 
Thanks for all your support.
Hello @vojtechjelinek 
I figured out that in my previous comment I accidentally mentioned `statistics/gae_models.py` instead of `base_model/gae_models.py`. Will you please correct it. I'm done with `statistics/gae_models.py` and waiting for your approval.
Thanks
@vojtechjelinek can I take up the two remaining files ```base_model/gae_models.py``` and ``` wipeout_services.py```? I was suggested to work on this issue as a part of my onboarding.
@OBITORASU Sure, I have assigned those to you.
@vojtechjelinek I had a doubt regarding ```base_model/gae_models.py```
So to elaborate, we have a classmethod namely:
``` 
@classmethod
    def delete_multi(cls, entities):
        """Deletes the given datastore_services.Model instances.

        Args:
            entities: list(datastore_services.Model). The list of model
                instances to be deleted.
        """
        datastore_services.delete_multi([entity.key for entity in entities])
```
Now datastore_services is defined as:
```
datastore_services = models.Registry.import_datastore_services()
```
So till here I have a clear idea of what is going on. Now the issue is when searching for ```transaction_services.run_in_transaction``` I came across this particular line of code:
```
transaction_services.run_in_transaction(
                    datastore_services.delete_multi,
                    all_models_keys[
                        i:i + feconf.MAX_NUMBER_OF_OPS_IN_TRANSACTION])
```
Now my question is why have a classmethod delete_multi if we are not using BaseModel.delete_multi anywhere in code of the particular file. As there has been no instance of BaseModel.delete_multi when I try to find it.

Images for reference:
![Screenshot_20210129_113536](https://user-images.githubusercontent.com/65222459/106238851-99e37780-6227-11eb-9dfe-96b1ce788e30.png)

![Screenshot_20210129_113552](https://user-images.githubusercontent.com/65222459/106238880-a962c080-6227-11eb-9f04-067932d5b068.png)

![Screenshot_20210129_113813](https://user-images.githubusercontent.com/65222459/106238901-b4b5ec00-6227-11eb-9a1a-2648b4ffee5e.png)

![image](https://user-images.githubusercontent.com/65222459/106238996-dfa04000-6227-11eb-905a-6f200387a8e5.png)




Also another question regarding the ```wipeout_services.py```
Just to confirm, is it ```wipeout_services.py``` or ```wipeout_service.py``` ?
Apparently I can't fuzzy find ```wipeout_services.py``` the closest file to it is ```wipeout_service.py``` which I can find.
@OBITORASU The `BaseModel.delete_multi` can be used in other files, also since some models inherit from `BaseModel` (like `UserSettingsModel`), the `delete_multi` can be used in these classes too.
Thanks for the explanation :)
@Nik-09 How is your work going? Do you need help with anything?
> @Nik-09 How is your work going? Do you need help with anything?

I have created Pr please ask @aks681 to review [pr link](https://github.com/oppia/oppia/pull/11781)
Thanks @vojtechjelinek and @aks681 .

Hi @vojtechjelinek , Is there any task available in this issue?
@17sushmita No, it isn't.

Resolving patch:
diff --git a/core/platform/transactions/gae_transaction_services.py b/core/platform/transactions/gae_transaction_services.py
--- a/core/platform/transactions/gae_transaction_services.py
+++ b/core/platform/transactions/gae_transaction_services.py
@@ -24,34 +24,6 @@
 from google.appengine.ext import ndb
 
 
-# TODO(#11431): Remove this in favor of run_in_transaction_wrapper.
-def run_in_transaction(fn, *args, **kwargs):
-    """Runs a function in a transaction. Either all of the operations in
-    the transaction are applied, or none of them are applied.
-
-    If an exception is raised, the transaction is likely not safe to
-    commit, since TransactionOptions.ALLOWED is used.
-
-    Args:
-        fn: callable. A function (or callable) to be called.
-        *args: list(*). Variable length argument list passed to the callable.
-        **kwargs: *. Arbitrary keyword arguments passed to the callable.
-
-    Returns:
-        *. Whatever fn() returns.
-
-    Raises:
-        Exception. Whatever fn() raises.
-        datastore_errors.TransactionFailedError. The transaction failed.
-    """
-    # XG enables cross-group transaction that can handle up to 25 model groups.
-    return ndb.transaction(
-        lambda: fn(*args, **kwargs),
-        xg=True,
-        propagation=ndb.TransactionOptions.ALLOWED,
-    )
-
-
 def run_in_transaction_wrapper(fn):
     """Runs a decorated function in a transaction. Either all of the operations
     in the transaction are applied, or none of them are applied.
@@ -69,7 +41,11 @@ def run_in_transaction_wrapper(fn):
     @functools.wraps(fn)
     def wrapper(*args, **kwargs):
         """Wrapper for the transaction."""
-        return run_in_transaction(fn, *args, **kwargs)
+        return ndb.transaction(
+            lambda: fn(*args, **kwargs),
+            xg=True,
+            propagation=ndb.TransactionOptions.ALLOWED,
+        )
 
     return wrapper
 
diff --git a/core/storage/collection/gae_models.py b/core/storage/collection/gae_models.py
--- a/core/storage/collection/gae_models.py
+++ b/core/storage/collection/gae_models.py
@@ -180,9 +180,6 @@ def _trusted_commit(
 
         collection_rights = CollectionRightsModel.get_by_id(self.id)
 
-        # TODO(msl): Test if put_async() leads to any problems (make
-        # sure summary dicts get updated correctly when collections
-        # are changed).
         collection_commit_log = CollectionCommitLogEntryModel.create(
             self.id, self.version, committer_id, commit_type, commit_message,
             commit_cmds, collection_rights.status,
diff --git a/core/storage/exploration/gae_models.py b/core/storage/exploration/gae_models.py
--- a/core/storage/exploration/gae_models.py
+++ b/core/storage/exploration/gae_models.py
@@ -173,9 +173,6 @@ def _trusted_commit(
 
         exp_rights = ExplorationRightsModel.get_by_id(self.id)
 
-        # TODO(msl): Test if put_async() leads to any problems (make
-        # sure summary dicts get updated correctly when explorations
-        # are changed).
         exploration_commit_log = ExplorationCommitLogEntryModel.create(
             self.id, self.version, committer_id, commit_type, commit_message,
             commit_cmds, exp_rights.status, exp_rights.community_owned


Test patch:
diff --git a/core/domain/activity_jobs_one_off_test.py b/core/domain/activity_jobs_one_off_test.py
--- a/core/domain/activity_jobs_one_off_test.py
+++ b/core/domain/activity_jobs_one_off_test.py
@@ -571,9 +571,6 @@ def _trusted_commit(
         # Create and delete events will already be recorded in the
         # CollectionModel.
         if commit_type not in ['create', 'delete']:
-            # TODO(msl): Test if put_async() leads to any problems (make
-            # sure summary dicts get updated correctly when collections
-            # are changed).
             collection_models.CollectionCommitLogEntryModel(
                 id=('rights-%s-%s' % (self.id, self.version)),
                 user_id=committer_id,
@@ -618,9 +615,6 @@ def _trusted_commit(
         # Create and delete events will already be recorded in the
         # ExplorationModel.
         if commit_type not in ['create', 'delete']:
-            # TODO(msl): Test if put_async() leads to any problems (make
-            # sure summary dicts get updated correctly when explorations
-            # are changed).
             exp_models.ExplorationCommitLogEntryModel(
                 id=('rights-%s-%s' % (self.id, self.version)),
                 user_id=committer_id,
diff --git a/core/jobs_test.py b/core/jobs_test.py
--- a/core/jobs_test.py
+++ b/core/jobs_test.py
@@ -703,6 +703,7 @@ def _handle_incoming_event(
             unused_state_name, unused_session_id, unused_params,
             unused_play_type):
 
+        @transaction_services.run_in_transaction_wrapper
         def _increment_counter():
             """Increments the count, if the realtime model corresponding to the
             active real-time model id exists.
@@ -715,7 +716,7 @@ def _increment_counter():
                 id=realtime_model_id, count=1,
                 realtime_layer=active_realtime_layer).put()
 
-        transaction_services.run_in_transaction(_increment_counter)
+        _increment_counter()
 
     # Public query method.
     @classmethod