Repository: inventree/InvenTree

Issue:
[BUG] could not serialize access due to concurrent update
could not serialize access due to concurrent update

and just crash....


Discussion:
@batara666 are you using an sqlite database? The concurrency limits there may be causing trouble.

Can you post an error message?
postgresql
hi @batara666 . To enable diagnosis we need a bit more info. Could you provide the following:

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Deployment Method**
Docker
Bare Metal

**Version Information**
You can get this by going to the "About InvenTree" section in the upper right corner and clicking on to the "copy version information"
I'm facing the same error, installing a new instance.
I've follow requirements as python virtual env, postgresql, etc. I'm installing in a bare metal server and just after "invoke update" and "invoke superuser" the guide let you install as production server.

In my case the error appears as I'm trying to start gunicorn from

 (env) inventree@server:~/src/InvenTree$ /opt/inventree/env/bin/gunicorn -c gunicorn.conf.py InvenTree.wsgi -b 0.0.0.0:8000

/opt/inventree is the home directory of "inventree" linux user
I've postgresql (12) installed and all django migrations were ok
attached you'll find the complete trace

PS: it's correct to install Inventree from a git pull from MASTER branch rather than STABLE one? (as of https://inventree.readthedocs.io/en/latest/start/install/#download-source-code)

[trace.log](https://github.com/inventree/InvenTree/files/7475765/trace.log)

 
> I'm facing the same error, installing a new instance.
> 
> I've follow requirements as python virtual env, postgresql, etc. I'm installing in a bare metal server and just after "invoke update" and "invoke superuser" the guide let you install as production server.
> 
> 
What base OS are you using? if possible maybe just provide the Uname.
> 
> In my case the error appears as I'm trying to start gunicorn from
> 
> 
> 
>  (env) inventree@server:~/src/InvenTree$ /opt/inventree/env/bin/gunicorn -c gunicorn.conf.py InvenTree.wsgi -b 0.0.0.0:8000
> 
> 
> 
> /opt/inventree is the home directory of "inventree" linux user
> 
> I've postgresql (12) installed and all django migrations were ok
> 
> attached you'll find the complete trace
> 
> 
can you please rerun the migrations and check if they error out now?
> 
> PS: it's correct to install Inventree from a git pull from MASTER branch rather than STABLE one? (as of https://inventree.readthedocs.io/en/latest/start/install/#download-source-code)
> 
For production use we recommend stable, although the difference is small normally. Please try with stable for now as the upcoming release is quite big.


> What base OS are you using? if possible maybe just provide the Uname.
**Ubuntu 20.04 LTS, no pending update with 5.4.0-89-generic kernel**

> can you please rerun the migrations and check if they error out now?
**no different behaviour, same errors as before**

> For production use we recommend stable, although the difference is small normally. Please try with stable for now as the upcoming release is quite big.

I've dropped the entire db and "src" folder, cloned the repo (**"stable" branch**), created the db and re-apply migrations and superuser.

Now the gunicorn seems to work as expected:

(env) inventree@server:~/src/InvenTree$ /opt/inventree/env/bin/gunicorn -c gunicorn.conf.py InvenTree.wsgi -b 0.0.0.0:8000
[2021-11-04 16:49:34 +0100] [18756] [INFO] Starting gunicorn 20.1.0
[2021-11-04 16:49:34 +0100] [18756] [INFO] Listening at: http://0.0.0.0:8000 (18756)
[2021-11-04 16:49:34 +0100] [18756] [INFO] Using worker: sync
[2021-11-04 16:49:34 +0100] [18758] [INFO] Booting worker with pid: 18758
[2021-11-04 16:49:34 +0100] [18759] [INFO] Booting worker with pid: 18759
[2021-11-04 16:49:34 +0100] [18760] [INFO] Booting worker with pid: 18760
[2021-11-04 16:49:34 +0100] [18761] [INFO] Booting worker with pid: 18761
[2021-11-04 16:49:35 +0100] [18762] [INFO] Booting worker with pid: 18762
[2021-11-04 16:49:35 +0100] [18763] [INFO] Booting worker with pid: 18763
[2021-11-04 16:49:35 +0100] [18764] [INFO] Booting worker with pid: 18764
[2021-11-04 16:49:35 +0100] [18765] [INFO] Booting worker with pid: 18765
[2021-11-04 16:49:35 +0100] [18766] [INFO] Booting worker with pid: 18766
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'
Base currency changed from USD to EUR
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'EUR'

So it seems that with the stable branch everything is working good, maybe it worth a while to put in the docs that "stable" branch is the recommend one?

Thank you in advance!
@argaar ok good to know this works on stable. We will have to look at this before the next major release.

@SchrodingersGat this log seems to show the problem with the exchange rates updating multiple times again.
@matmair I'd like to keep this one open, it seems like a legitimate issue.

@nwns you recently made some changes to the postgresql default configuration options. Do you think that might have caused some issues here?
> @matmair I'd like to keep this one open, it seems like a legitimate issue.
> 
> 
> 
> @nwns you recently made some changes to the postgresql default configuration options. Do you think that might have caused some issues here?

I tried to replicate and everything seems to work on pgsql 14 and current versions of patched pgsql 12. The default settings for pgsql 12 used to make similar issues but that was solved long ago.

I would strongly suggest to use pgsql 14 in general
My error version cause maybe because I suddenly press ctrl+c the webserver, when I ran the server again, It's always erroring and the app can't recover it-self from that kind of error
@batara666 what version of postgresql are you running on your setup?
@SchrodingersGat Yes, the exception being logged would be the result of the change to serializable for postgres.

Normally the correct answer is to catch the OperationalError and try the transaction again, however, looking at the call stack (but not the code around it), this is probably a bug in the server startup code where it has multiple outbound calls to update something to do with currency because it is exactly the same code path in all three workers that crashed.

##### Speculation
Because of how this exception works, there is probably a 4th worker that was first and was the one that actually updated the table to cause the serialization failure
@SchrodingersGat how should we move forward with this?
I'd like to have a "fix" by `0.6.0` but I'm not sure what the problem is. It does not seem to occur in the unit tests, and the [demo database](https://demo.inventree.org) is running on postgresql without issue.

Is it as simple as mandating a sufficiently new version of postgresql? @nwns is this accurate?
@SchrodingersGat I am pretty sure we are seeing a race condition that is being highlighted by the change in the transaction behaviour as I outlined above.

Our current installs of Inventree (using Postgres 10) do not show this issue, but they are also older versions of Inventree.  I have not yet got the current version up on a cluster yet.
I did a bare metal install on Ubuntu 21.04 server w/PostgreSQL 14, MASTER branch. Changed the default currency to EUR in config.yaml.

The same issue as described by @batara666 appeared when starting gunicorn: `django.db.utils.OperationalError: could not serialize access due to concurrent update`.

However, when I changed the default currency back to USD, gunicorn started without problems.
I have looked into this.  As the [Django docs](https://docs.djangoproject.com/en/3.2/ref/applications/#django.apps.AppConfig.ready) say, we should not be interacting with the database in App Ready. It is for setting up signals not for fiddling with the database.

This currency data update should be moved to an async task. Since we now have background workers, it should probably be done by them. Maybe spawning a thread would work but it would have the same issue, as would just catching the database concurrency exception and running the update again.

I will look into what it would take later today.
@nwns there is a lot of discussion about what should take place in ready in the community. If you look at core code ready is called after all models are ready so it is safe. Problems arise if you modify models in the ready code - which a lot of projects do. We don't in general.

Moving the code could have bad consequences in business environments. Prices for parts could change mid-process as the conversion rates change.
I think the best way to solve this is just a flag that is checked as this would make sure that the task is not already running. A bigger refactor should be discussed for its consequences.
> Moving the code could have bad consequences in business environments. Prices for parts could change mid-process as the conversion rates change.

@matmair Sure, but that can already happen when the scheduled task to update the exchange ranges triggers every day.  That is the problem with live exchange rates in general.  This is why businesses hedge the exchange rate and fix it for a certain amount of time.

Perhaps the actual solution to this, it to have a button in Admin for the user to click and update the exchange rates when they want.  Possibly with instructions on how to schedule an exchange rate update if they don't mind the rates changing whenever.


> I think the best way to solve this is just a flag that is checked as this would make sure that the task is not already running. A bigger refactor should be discussed for its consequences.

The quick fix is to to just catch the exception and retry the update.

Resolving patch:
diff --git a/InvenTree/InvenTree/exchange.py b/InvenTree/InvenTree/exchange.py
--- a/InvenTree/InvenTree/exchange.py
+++ b/InvenTree/InvenTree/exchange.py
@@ -2,6 +2,7 @@
 from urllib.error import HTTPError, URLError
 
 from djmoney.contrib.exchange.backends.base import SimpleExchangeBackend
+from django.db.utils import OperationalError
 
 
 class InvenTreeExchange(SimpleExchangeBackend):
@@ -32,3 +33,12 @@ def update_rates(self, base_currency=currency_code_default()):
         # catch connection errors
         except (HTTPError, URLError):
             print('Encountered connection error while updating')
+        except OperationalError as e:
+            if 'SerializationFailure' in e.__cause__.__class__.__name__:
+                print('Serialization Failure while updating exchange rates')
+                # We are just going to swallow this exception because the
+                # exchange rates will be updated later by the scheduled task
+            else:
+                # Other operational errors probably are still show stoppers
+                # so reraise them so that the log contains the stacktrace
+                raise


Test patch: