Repository: apache/airflow

Issue:
Bad terminal handling of `airflow db shell` command with Postgres DB
### Apache Airflow version

main (development)

### What happened

This happens in both iTerm 2 and the built-in Terminal.app in macOS when running ~breeze start-airflow~. `airflow db shell` when using postgres as a backend.

Within `psql` (run via `airflow db shell`):

* Scrollback editing (eg: hitting up arrow to edit a previous query) is messed up for multi-line queries. I don't have a better description. I recorded a video of what happens.
* Some things require an additional keypress (eg: quitting the pager)
  - Some things, like ctrl+P to scrollback needing an additional keypress happen in a normal Docker environment. This is different and distinct from that.
* SIGINT isn't registered when trying to cancel editing a query and reset the SQL prompt

I suspect that there's a lot of interaction happening between iTerm/Terminal.app, ZSH, Python/Breeze, Bash, Docker, tmux, Python/Airflow CLI, and psql, and some keypresses are being filtered or queued along the way. So there are multiple fun rabbit holes to chase down to solve this.

### What you think should happen instead

* Scrollback editing should work. Your text cursor should be in the correct place and the text should be rendered correctly.
* An additional keypress to exit the pager should not be necessary
* SIGINT should cancel editing the query and reset the edit line (eg: it should return to an empty SQL prompt: `airflow=#` instead of `airflow-#`)

### How to reproduce

* Scrollback editing is messed up - indentation and text wrapping are off/wrong
  - This happens even when I don't adjust the tmux pane sizes at all
  - Example:
    + Run `breeze start-airflow` in a terminal 
    + Run `docker exec -it $CONTAINER_ID /bin/bash` with the breeze container that is running
    + Run `airflow db shell` within the container
    + Then this happens (note the up arrow keypresses aren't called out visually):

      https://user-images.githubusercontent.com/597113/188750191-2b3e8ef3-ee89-4b2e-872d-1451807bdbc7.mov

* Some things require an additional keypress
  Example:
  - Run a SQL query:
      ```
      airflow=# SELECT * FROM dag_schedule_dataset_reference
      ```
  - it displays in the pager
      ```
       dataset_id |        dag_id         |          created_at           |          updated_at
      ------------+-----------------------+-------------------------------+-------------------------------
                3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
                2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
                4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
                5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
                6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
                1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
      (6 rows)
      
      (END)
      ```
  - Usually you just hit `q` and it exits immediately, but the pager sticks around:
      ```
       dataset_id |        dag_id         |          created_at           |          updated_at
      ------------+-----------------------+-------------------------------+-------------------------------
                3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
                2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
                4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
                5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
                6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
                1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
      (6 rows)
      
      (END)
      ```
  - The `q` keypress is registered upon your next keypress, whatever that is (in this case, `6`). And the second keypress is echoed in the SQL editor:
      ```
      airflow=# SELECT * FROM dag_schedule_dataset_reference;
      airflow=# 6
      ```
* SIGINT isn't registered to cancel editing a query
  Example:
  - Write a query:
      ```
      airflow=# SELECT * FROM dag_messedupspelling
      ```
  - Then try to cancel/reset it with Ctrl+C:
      ```
      airflow=# SELECT * FROM dag_messedupspelling
      airflow-#
      ```
      (nothing happens)
      In normal psql this will reset the query editor (you can tell because the `airflow-#` prompt changes back to `airflow=#`).


### Operating System

macOS 11.6.8 Big Sur

### Versions of Apache Airflow Providers

N/A

### Deployment

Other Docker-based deployment

### Deployment details

Breeze

### Anything else

This happens every time.

Apologies if this isn't the place to report Breeze issues.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
This is from one pane, right? I wonder if this is a Tmux issue (and if that’s the case there’s not much we can do). Can you try entering the shell (with `breeze shell` or even plain `docker run`) and run `tmux` manually, and see if this happens to you as well?

(This could also be related to terminal configuration; terminal emulation is pesky work.)
This is neither breeze nor tmux problem. This is problem with `airflow db shell` for postgres and pass-through to `pysql` which should likely be improved. 

You got the exact same behaviour when you use local virtualenv without either of the breeze/tmux (but you can reproduce it easily using breeze as the postgres database setup:

1. Create airflow virtualenv (an easy way is to activate empty venv and use ./scripts/tools/initialize_virtualenv.py - it will use constraints and you can choose which extras to install)
2. run `breeze --backend postgres --db-reset` (this will start and reset the postgres DB as well)
3. exit breeze
4. Setup your local venv airflow to connect to the Postgres db of Breeze:
```
export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:airflow@127.0.0.1:25433/airflow
```
(you can see Postgres port number user/pasword used printed in the cheathsheet when you start breeze or find it in the docs)
5) Run `airflow db shell`

Observe the very same behaviour.

I updated the description to clarify it. Fancy to attempt to fix it @blag ? I think it is the matter of how subprocess is used to run `psql` 
BTW. I jsut tested it additionally running this in `start-airflow` pane:
```
pysql -h postgres -U postgres
```
(password airflow)
Which is equivalent of what `breeze db shell` does for postgres, 

And it behaves as expected - so this is definitely `airflow db shell` that introduces it
Tangentally I’m wondering if it’s a good idea to use [dbcli tools](https://github.com/dbcli) for `airflow db shell`.
Looks cool. Definitely worth trying how well it works in airflow db. It will likely increase the dependency size, so possibly it should be something that is installed as "extra" ? 
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
❌ Still broken - Scrollback editing is messed up - indentation and text wrapping are off/wrong
✅ Fixed...? - Some things require an additional keypress
❌ Still broken - SIGINT isn't registered to cancel editing a query
Sure. Marked it as good first issue - maybe you would like to fix it @blag ? Otherwise it will have to wait for someone who would like to fix it.
I can fix the SIGINT issue. I think the problem is in the fact that the TTY is set to raw mode, so in order to send the SIGINT signal to the subprocess, you need to listen for character code 3 in the input data.

The other issues mentioned here are quite puzzling for me though.
If y ou can move it forward, that would be cool.

Resolving patch:
diff --git a/airflow/utils/process_utils.py b/airflow/utils/process_utils.py
--- a/airflow/utils/process_utils.py
+++ b/airflow/utils/process_utils.py
@@ -205,12 +205,12 @@ def execute_interactive(cmd: list[str], **kwargs) -> None:
     log.info("Executing cmd: %s", " ".join(shlex.quote(c) for c in cmd))
 
     old_tty = termios.tcgetattr(sys.stdin)
-    tty.setraw(sys.stdin.fileno())
+    old_sigint_handler = signal.getsignal(signal.SIGINT)
+    tty.setcbreak(sys.stdin.fileno())
 
     # open pseudo-terminal to interact with subprocess
     primary_fd, secondary_fd = pty.openpty()
     try:
-        # use os.setsid() make it run in a new process group, or bash job control will not be enabled
         with subprocess.Popen(
             cmd,
             stdin=secondary_fd,
@@ -219,6 +219,8 @@ def execute_interactive(cmd: list[str], **kwargs) -> None:
             universal_newlines=True,
             **kwargs,
         ) as proc:
+            # ignore SIGINT in the parent process
+            signal.signal(signal.SIGINT, signal.SIG_IGN)
             while proc.poll() is None:
                 readable_fbs, _, _ = select.select([sys.stdin, primary_fd], [], [])
                 if sys.stdin in readable_fbs:
@@ -230,6 +232,7 @@ def execute_interactive(cmd: list[str], **kwargs) -> None:
                         os.write(sys.stdout.fileno(), output_data)
     finally:
         # restore tty settings back
+        signal.signal(signal.SIGINT, old_sigint_handler)
         termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_tty)
 
 


Test patch: