Repository: apache/airflow

Issue:
Random "duplicate key value violates unique constraint" errors when initializing the postgres database
### Apache Airflow version

2.3.0 (latest released)

### What happened

while testing airflow 2.3.0 locally (using postgresql 12.4), the webserver container shows random errors:
```
webserver_1  | + airflow db init
...
webserver_1  | + exec airflow webserver
...
webserver_1  | [2022-05-04 18:58:46,011] {{manager.py:568}} INFO - Added Permission menu access on Permissions to role Admin
postgres_1   | 2022-05-04 18:58:46.013 UTC [41] ERROR:  duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
postgres_1   | 2022-05-04 18:58:46.013 UTC [41] DETAIL:  Key (permission_view_id, role_id)=(204, 1) already exists.
postgres_1   | 2022-05-04 18:58:46.013 UTC [41] STATEMENT:  INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), 204, 1) RETURNING ab_permission_view_role.id
webserver_1  | [2022-05-04 18:58:46,015] {{manager.py:570}} ERROR - Add Permission to Role Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
webserver_1  | DETAIL:  Key (permission_view_id, role_id)=(204, 1) already exists.
webserver_1  |
webserver_1  | [SQL: INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
webserver_1  | [parameters: {'permission_view_id': 204, 'role_id': 1}]
```

notes:
1. when the db is first initialized, i have ~40 errors like this (with ~40 different `permission_view_id` but always the same `'role_id': 1`)
2. when it's not the first time initializing db, i always have 1 error like this but it shows different `permission_view_id` each time
3. all these errors don't seem to have any real negative effects, the webserver is still running and airflow is still running and scheduling tasks
4. "occasionally" i do get real exceptions which render the webserver workers all dead:

```
postgres_1   | 2022-05-05 20:03:30.580 UTC [44] ERROR:  duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
postgres_1   | 2022-05-05 20:03:30.580 UTC [44] DETAIL:  Key (permission_view_id, role_id)=(214, 1) already exists.
postgres_1   | 2022-05-05 20:03:30.580 UTC [44] STATEMENT:  INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), 214, 1) RETURNING ab_permission_view_role.id
webserver_1  | [2022-05-05 20:03:30 +0000] [121] [ERROR] Exception in worker process
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
webserver_1  |     self.dialect.do_execute(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
webserver_1  |     cursor.execute(statement, parameters)
webserver_1  | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
webserver_1  | DETAIL:  Key (permission_view_id, role_id)=(214, 1) already exists.
webserver_1  |
webserver_1  |
webserver_1  | The above exception was the direct cause of the following exception:
webserver_1  |
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
webserver_1  |     worker.init_process()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
webserver_1  |     self.load_wsgi()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
webserver_1  |     self.wsgi = self.app.wsgi()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
webserver_1  |     self.callable = self.load()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
webserver_1  |     return self.load_wsgiapp()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
webserver_1  |     return util.import_app(self.app_uri)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/util.py", line 412, in import_app
webserver_1  |     app = app(*args, **kwargs)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 158, in cached_app
webserver_1  |     app = create_app(config=config, testing=testing)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 146, in create_app
webserver_1  |     sync_appbuilder_roles(flask_app)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 68, in sync_appbuilder_roles
webserver_1  |     flask_app.appbuilder.sm.sync_roles()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 580, in sync_roles
webserver_1  |     self.update_admin_permission()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 562, in update_admin_permission
webserver_1  |     self.get_session.commit()
webserver_1  |   File "<string>", line 2, in commit
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1423, in commit
webserver_1  |     self._transaction.commit(_to_root=self.future)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829, in commit
webserver_1  |     self._prepare_impl()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
webserver_1  |     self.session.flush()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3255, in flush
webserver_1  |     self._flush(objects)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3395, in _flush
webserver_1  |     transaction.rollback(_capture_exception=True)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
webserver_1  |     compat.raise_(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
webserver_1  |     raise exception
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3355, in _flush
webserver_1  |     flush_context.execute()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 453, in execute
webserver_1  |     rec.execute(self)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 576, in execute
webserver_1  |     self.dependency_processor.process_saves(uow, states)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1182, in process_saves
webserver_1  |     self._run_crud(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1245, in _run_crud
webserver_1  |     connection.execute(statement, secondary_insert)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1200, in execute
webserver_1  |     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 313, in _execute_on_connection
webserver_1  |     return connection._execute_clauseelement(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1389, in _execute_clauseelement
webserver_1  |     ret = self._execute_context(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1748, in _execute_context
webserver_1  |     self._handle_dbapi_exception(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1929, in _handle_dbapi_exception
webserver_1  |     util.raise_(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
webserver_1  |     raise exception
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
webserver_1  |     self.dialect.do_execute(
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
webserver_1  |     cursor.execute(statement, parameters)
webserver_1  | sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
webserver_1  | DETAIL:  Key (permission_view_id, role_id)=(214, 1) already exists.
webserver_1  |
webserver_1  | [SQL: INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
webserver_1  | [parameters: {'permission_view_id': 214, 'role_id': 1}]
webserver_1  | (Background on this error at: http://sqlalche.me/e/14/gkpj)
webserver_1  | [2022-05-05 20:03:30 +0000] [121] [INFO] Worker exiting (pid: 121)
flower_1     | + exec airflow celery flower
scheduler_1  | + exec airflow scheduler
webserver_1  | [2022-05-05 20:03:31 +0000] [118] [INFO] Worker exiting (pid: 118)
webserver_1  | [2022-05-05 20:03:31 +0000] [119] [INFO] Worker exiting (pid: 119)
webserver_1  | [2022-05-05 20:03:31 +0000] [120] [INFO] Worker exiting (pid: 120)
worker_1     | + exec airflow celery worker
```
However such exceptions are rare and pure random, i can't find a way to reproduce them consistently.

### What you think should happen instead

prior to 2.3.0 there were no such errors 

### How to reproduce

_No response_

### Operating System

Linux Mint 20.3

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
@ephraimbuddy sorry man, i actually have no idea what could cause the issue. i have only wild guesses :sweat_smile: 
Oh, I assigned it to you because you checked that you are willing to submit a PR. I'll unassign you now :)
further investigation shows that we are removing the same "permissions" and "permission views" repeatedly and trying to add them back repeatedly:
```
webserver_1  | [2022-05-07 02:50:25 +0000] [107] [INFO] Using worker: sync
webserver_1  | [2022-05-07 02:50:25 +0000] [117] [INFO] Booting worker with pid: 117
webserver_1  | [2022-05-07 02:50:25 +0000] [118] [INFO] Booting worker with pid: 118
webserver_1  | [2022-05-07 02:50:25 +0000] [119] [INFO] Booting worker with pid: 119
webserver_1  | [2022-05-07 02:50:25 +0000] [120] [INFO] Booting worker with pid: 120
webserver_1  | [2022-05-07 02:50:25,341] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,400] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,532] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,600] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,692] {{manager.py:585}} INFO - Removed Permission can create on Users to role Admin
webserver_1  | [2022-05-07 02:50:25,705] {{manager.py:585}} INFO - Removed Permission can create on Users to role Admin
webserver_1  | [2022-05-07 02:50:25,724] {{manager.py:543}} INFO - Removed Permission View: can_create on Users
webserver_1  | /usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py:1461 SAWarning: DELETE statement on table 'ab_permission_view' expected to delete 1 row(s); 0 were matched.  Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
webserver_1  | [2022-05-07 02:50:25,736] {{manager.py:543}} INFO - Removed Permission View: can_create on Users
webserver_1  | [2022-05-07 02:50:25,912] {{manager.py:585}} INFO - Removed Permission menu access on Permissions to role Admin
webserver_1  | [2022-05-07 02:50:25,912] {{manager.py:585}} INFO - Removed Permission menu access on Permissions to role Admin
webserver_1  | [2022-05-07 02:50:25,923] {{manager.py:543}} INFO - Removed Permission View: menu_access on Permissions
webserver_1  | /usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py:1461 SAWarning: DELETE statement on table 'ab_permission_view' expected to delete 1 row(s); 0 were matched.  Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
webserver_1  | [2022-05-07 02:50:25,935] {{manager.py:543}} INFO - Removed Permission View: menu_access on Permissions
webserver_1  | [2022-05-07 02:50:26,016] {{manager.py:508}} INFO - Created Permission View: menu access on Permissions
webserver_1  | [2022-05-07 02:50:26,027] {{manager.py:568}} INFO - Added Permission menu access on Permissions to role Admin
webserver_1  | [2022-05-07 02:50:26,067] {{manager.py:585}} INFO - Removed Permission menu access on Permissions to role Admin
webserver_1  | [2022-05-07 02:50:26,091] {{manager.py:543}} INFO - Removed Permission View: menu_access on Permissions
postgres_1   | 2022-05-07 02:50:26.093 UTC [43] ERROR:  insert or update on table "ab_permission_view_role" violates foreign key constraint "ab_permission_view_role_permission_view_id_fkey"
postgres_1   | 2022-05-07 02:50:26.093 UTC [43] DETAIL:  Key (permission_view_id)=(220) is not present in table "ab_permission_view".
postgres_1   | 2022-05-07 02:50:26.093 UTC [43] STATEMENT:  INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), 220, 1) RETURNING ab_permission_view_role.id
webserver_1  | [2022-05-07 02:50:26,096] {{manager.py:570}} ERROR - Add Permission to Role Error: (psycopg2.errors.ForeignKeyViolation) insert or update on table "ab_permission_view_role" violates foreign key constraint "ab_permission_view_role_permission_view_id_fkey"
webserver_1  | DETAIL:  Key (permission_view_id)=(220) is not present in table "ab_permission_view".
webserver_1  |
webserver_1  | [SQL: INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
webserver_1  | [parameters: {'permission_view_id': 220, 'role_id': 1}]
webserver_1  | (Background on this error at: http://sqlalche.me/e/14/gkpj)
```
notice the 2 `SAWarning: ...` when we tried to `Removed Permission View: can_create on Users`. Apparently the same `DELETE` was executed twice.

I don't understand why we `Removed Permission menu access on Permissions from role Admin` and then immediately `Added Permission menu access on Permissions to role Admin` (the same with `can create on Users` permission) :thinking: 

The database screenshots which shows the permissions and permission views are being deleted and added back hence their IDs are increasing

![2022-05-06_23-55](https://user-images.githubusercontent.com/14293802/167237280-a63d20ce-fbcf-4a35-af09-e67a838c5f18.png)

![2022-05-06_23-56](https://user-images.githubusercontent.com/14293802/167237282-219b3e79-2303-480d-b895-cef73d7450f6.png)

interestingly previous airflow versions also have the same behavior but there were no errors  `¯\_(ツ)_/¯`
This sounds like a bug in sync_appbuilder_roles implementation - when we have more than 1 webserver or when they are crashing during first start they then subsequently fail on starting another webserver via wsgi. @jhtimmins - maybe you can take a look at that ?
```
webserver_1  | [2022-05-07 02:50:25 +0000] [107] [INFO] Using worker: sync
webserver_1  | [2022-05-07 02:50:25 +0000] [117] [INFO] Booting worker with pid: 117
webserver_1  | [2022-05-07 02:50:25 +0000] [118] [INFO] Booting worker with pid: 118
webserver_1  | [2022-05-07 02:50:25 +0000] [119] [INFO] Booting worker with pid: 119
webserver_1  | [2022-05-07 02:50:25 +0000] [120] [INFO] Booting worker with pid: 120
webserver_1  | [2022-05-07 02:50:25,341] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,400] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,532] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,600] {{logging_config.py:46}} INFO - Successfully imported user-defined logging config from log_config.LOGGING_CONFIG
webserver_1  | [2022-05-07 02:50:25,692] {{manager.py:585}} INFO - Removed Permission can create on Users to role Admin
```
i see those repeated log messages right after booting up 4 worker sub-processes, are these workers trying to update the database at the same time using the same `DELETE` or `INSERT` statements?
Yeah. I think the problem is that the webserver spawns 4 parallel webserver processes (workers) and each process tries to update the permissions at the same time)
Which library I should downgrade to get my upgrade going. Any suggestions, please?
I don't think downgrade library will work? Seems like we have to wait for v2.3.1.

I am also facing exact thing for v2.3.0 using docker-compose installation
both
https://github.com/apache/airflow/blob/ee100a592e07761f3b32294f3ad82c4a6c7cf74d/airflow/www/app.py#L136
and
https://github.com/apache/airflow/blob/ee100a592e07761f3b32294f3ad82c4a6c7cf74d/airflow/www/app.py#L146
are doing clashing reads/writes to the database once this line
https://github.com/apache/airflow/blob/ee100a592e07761f3b32294f3ad82c4a6c7cf74d/airflow/cli/commands/webserver_command.py#L475
spins up 4 (by default) gunicorn workers. so i guess if i do this
```python
            os.environ['SKIP_DAGS_PARSING'] = 'True'
            app = create_app(None)
            os.environ.pop('SKIP_DAGS_PARSING')

            # set webserver.update_fab_perms to "False" here 
            # would cause the workers not to do the clashing reads/writes

            with subprocess.Popen(run_args, close_fds=True) as gunicorn_master_proc:
                monitor_gunicorn(gunicorn_master_proc.pid)
```
i know it's weird to alter it on the fly. an alternative is to set an env var such as
```python
os.environ['UPDATE_FAB_PERMS_DONE'] = 'True'
```
and then check it along side with `webserver.update_fab_perms` in https://github.com/apache/airflow/blob/ee100a592e07761f3b32294f3ad82c4a6c7cf74d/airflow/www/app.py#L67 and https://github.com/apache/airflow/blob/ee100a592e07761f3b32294f3ad82c4a6c7cf74d/airflow/www/extensions/init_appbuilder.py#L643

i tried the env var solution, it worked. but it's janky. it requires an extra env var :thinking: 
This seems somewhat related to #23926. cc @jhtimmins 
aha, i think it's exactly the same as https://github.com/apache/airflow/issues/23926: all gunicorn workers are executing the same code causing clashes on the database

maybe i'll move my comments there

Resolving patch:
diff --git a/airflow/www/fab_security/manager.py b/airflow/www/fab_security/manager.py
--- a/airflow/www/fab_security/manager.py
+++ b/airflow/www/fab_security/manager.py
@@ -787,7 +787,7 @@ def register_views(self):
         if self.appbuilder.app.config.get("FAB_ADD_SECURITY_PERMISSION_VIEWS_VIEW", True):
             self.appbuilder.add_view(
                 self.permissionmodelview,
-                "Permissions",
+                "Permission Pairs",
                 icon="fa-link",
                 label=_("Permissions"),
                 category="Security",


Test patch: