Repository: apache/airflow

Issue:
Increase connections HTTP login length to 5000 characters
### Description

The current length limit for the `login` parameter in an HTTP connection is 500 characters. It'd be nice if this was 5000 characters like the `password` parameter.

### Use case/motivation

We've run into an issue with an API we need to integrate with. It uses basic HTTP authentication, and both username and password are about 900 characters long each. We don't have any control over this API, so we cannot change the authentication method, nor the length of these usernames and passwords.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Do you refer to [airflow/models/connection.py](https://github.com/apache/airflow/blob/main/airflow/models/connection.py#L94) ?
I am not sure if this can be generally be extended as there might be limitations in the supported DB engines about maximum tuple sizes supported. But I am not 100% sure.

MS SQL can extend DB Pages, MySQL is limited to maximum 64KB but the data storage is based on the DB block size. For Postgres I see a hard limit following the documentation in https://www.postgresql.org/docs/current/limits.html: `The maximum number of columns for a table is further reduced as the tuple being stored must fit in a single 8192-byte heap page.`

So a login length of 1000 might still work but bumping the limit to 5000 chars (don't forget there is Unicode!) might produce DB errors.
> Do you refer to [airflow/models/connection.py](https://github.com/apache/airflow/blob/main/airflow/models/connection.py#L94) ? I am not sure if this can be generally be extended as there might be limitations in the supported DB engines about maximum tuple sizes supported. But I am not 100% sure.

Yep, exactly

> MS SQL can extend DB Pages, MySQL is limited to maximum 64KB but the data storage is based on the DB block size. For Postgres I see a hard limit following the documentation in https://www.postgresql.org/docs/current/limits.html: `The maximum number of columns for a table is further reduced as the tuple being stored must fit in a single 8192-byte heap page.`
> 
> So a login length of 1000 might still work but bumping the limit to 5000 chars (don't forget there is Unicode!) might produce DB errors.

I see what you mean, `password` is already 5000, plus another 5000 of `extra`. Assuming worst case scenario of 4 bytes per Unicode character, that'd be 20KB for each of those two columns. So 40 of 64. That being said, on the PostgreSQL documentation you link, on the next line, they mention:

> Variable-length fields of types such as text, varchar, and char can have their values stored out of line in the table's TOAST table when the values are large enough to require it. Only an 18-byte pointer must remain inside the tuple in the table's heap. For shorter length variable-length fields, either a 4-byte or 1-byte field header is used and the value is stored inside the heap tuple.

So potentially variable length columns don't count against the 64KB limit?
> > Variable-length fields of types such as text, varchar, and char can have their values stored out of line in the table's TOAST table when the values are large enough to require it. Only an 18-byte pointer must remain inside the tuple in the table's heap. For shorter length variable-length fields, either a 4-byte or 1-byte field header is used and the value is stored inside the heap tuple.
> 
> So potentially variable length columns don't count against the 64KB limit?

I only know from MSSQL that all variable pieces need to fir into one DB page. So in this case a longer password would be accepted if user name is short or the other way around. But nowadays MSSQL stores too long text values with a pointer to a separate page. But every DB engine has different limits.
@jens-scheffler-bosch Are there integration tests against all the different DB engines supported by Airflow? If so, it should be fairly easy to test out. Try to apply the migration making this change against all DB engines. If it works, great. If not, at least we know.
I have bad experiences in getting longer columns. For one MySQL has (very low) limit on index size - and if login would be indexed then it would fail miserably. I looked briefly and I think there is no index there so fundamentally this should be no problem. There is very low number of rows in such tables so I do not think this is really a problem.

I think in general  @Sighery you are right. If you want to check if it is ok - make a PR that changes it - including generating a migration, make a comment there to enable all tests (We will have to set "full tests needed" on such PR and we will see if current databases are ok with such change. 

If you look for history of changes for that file in GitHub, you will likely find a previous PR increasing the size of password field so you can take it as a base for the new PR.

Also some testing will have to be done on the UI side of things and API. So tests will have to likely be added to test the behaviour of those: including some screenshots from the UI when editing and possibly automated tests where such long passwords would be submitted via API/retrieved.

Feel free to try it @Sighery 

@jens-scheffler-bosch @potiuk So, tried it today [in my fork](https://github.com/Sighery/airflow/pull/1). I think I managed to set up the required CI checks correctly. I couldn't get Breeze to work locally (using NixOS which is a fair bit different to most other Linux distros), so I had to settle for the CI runs.

[MySQL is indeed failing with the migrations](https://github.com/Sighery/airflow/actions/runs/5644920445/job/15290684879?pr=1#step:5:1735). But it's strange, it actually fails when downgrading. And it fails on downgrading [migration 0079](https://github.com/apache/airflow/blob/d05e42e5d2081909c9c33de4bd4dfb759ac860c1/airflow/migrations/versions/0079_2_0_2_increase_size_of_connection_extra_field_.py#L4). This was the `extra` field in connection that used to be a `5000` characters `VARCHAR`. Migration 79 converted it to `TEXT`. The downgrade then tries to convert it back into `VARCHAR(5000)` and fails.

Not quite sure why this happens only after applying `VARCHAR(5000)` change to `login`. The downgrade of the `login` migration should also bring it back down to `VARCHAR(500)`, unless I messed up somewhere. If I didn't mess up the downgrade code, I'm guessing perhaps downgrading length afterwards doesn't decrease/release the page size, at least not in MySQL.
I can make the PR against this repository if it's any easier for testing as well. Just let me know.
Thanks for spending the effort. Sounds interesting.
I think opening a PR towards apache/airflow repo will be best, then contributors can check code and CI results.
Opened the PR against this repo at #32815, although I'm fairly certain it'll fail here as well. But that should be fine. At least I can get comments and opinions by other people who are more familiar with the codebase than me 👍 

Resolving patch:
diff --git a/airflow/migrations/versions/0131_2_8_0_make_connection_login_password_text.py b/airflow/migrations/versions/0131_2_8_0_make_connection_login_password_text.py
new file mode 100644
--- /dev/null
+++ b/airflow/migrations/versions/0131_2_8_0_make_connection_login_password_text.py
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Make connection login/password TEXT
+
+Revision ID: bd5dfbe21f88
+Revises: f7bf2a57d0a6
+Create Date: 2023-09-14 17:16:24.942390
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = "bd5dfbe21f88"
+down_revision = "f7bf2a57d0a6"
+branch_labels = None
+depends_on = None
+airflow_version = "2.8.0"
+
+
+def upgrade():
+    """Apply Make connection login/password TEXT"""
+    with op.batch_alter_table("connection", schema=None) as batch_op:
+        batch_op.alter_column(
+            "login", existing_type=sa.VARCHAR(length=500), type_=sa.Text(), existing_nullable=True
+        )
+        batch_op.alter_column(
+            "password", existing_type=sa.VARCHAR(length=5000), type_=sa.Text(), existing_nullable=True
+        )
+
+
+def downgrade():
+    """Unapply Make connection login/password TEXT"""
+    with op.batch_alter_table("connection", schema=None) as batch_op:
+        batch_op.alter_column(
+            "password", existing_type=sa.Text(), type_=sa.VARCHAR(length=5000), existing_nullable=True
+        )
+        batch_op.alter_column(
+            "login", existing_type=sa.Text(), type_=sa.VARCHAR(length=500), existing_nullable=True
+        )
diff --git a/airflow/models/connection.py b/airflow/models/connection.py
--- a/airflow/models/connection.py
+++ b/airflow/models/connection.py
@@ -93,8 +93,8 @@ class Connection(Base, LoggingMixin):
     description = Column(Text().with_variant(Text(5000), "mysql").with_variant(String(5000), "sqlite"))
     host = Column(String(500))
     schema = Column(String(500))
-    login = Column(String(500))
-    _password = Column("password", String(5000))
+    login = Column(Text())
+    _password = Column("password", Text())
     port = Column(Integer())
     is_encrypted = Column(Boolean, unique=False, default=False)
     is_extra_encrypted = Column(Boolean, unique=False, default=False)
diff --git a/airflow/utils/db.py b/airflow/utils/db.py
--- a/airflow/utils/db.py
+++ b/airflow/utils/db.py
@@ -88,7 +88,7 @@
     "2.6.0": "98ae134e6fff",
     "2.6.2": "c804e5c76e3e",
     "2.7.0": "405de8318b3a",
-    "2.8.0": "f7bf2a57d0a6",
+    "2.8.0": "bd5dfbe21f88",
 }
 
 


Test patch: