Issue 1:
Apache Airflow version

2.3.0 (latest released)

What happened

The unique ID of the task is seemingly not calculated correctly in base.py and it can cause in a duplicate key error, my guess is that this bit from /decorators/base.py:115:

if tg_task_id not in dag.task_ids:
return task_id

doesn't make any sense and the actual return should be:

if tg_task_id not in dag.task_ids:
return tg_task_id

But I didn't write it so there is maybe a reason down the line for return task_id. Someone that does know his way around TaskFlow's codebase should pitch in.

What you think should happen instead

task.expand should not lead to a airflow.exceptions.DuplicateTaskIdFound inside task groups.

How to reproduce

Pick example_task_group_decorator.py, change it so that task_1 returns a list, and call task_2 with .expand(value=task_1).

A modification as simple as this will already kill the DAG:

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

"""Example DAG demonstrating the usage of the @taskgroup decorator."""

import pendulum

from airflow.decorators import task, task_group
from airflow.models.dag import DAG

# [START howto_task_group_decorator]
# Creating Tasks
@task
def task_start():
"""Empty Task which is First Task of Dag"""
return '[Task_start]'

@task
def task_1(value: int) -> list[str]:
"""Empty Task1"""
return [f'[ Task1 {value} ]', f'[ Task1 {value+1} ]']

@task
def task_2(value: list[str]) -> str:
"""Empty Task2"""
return f'[ Task2 {value} ]'

@task
def task_3(value: str) -> None:
"""Empty Task3"""
print(f'[ Task3 {value} ]')

@task
def task_end() -> None:
"""Empty Task which is Last Task of Dag"""
print('[ Task_End ]')

# Creating TaskGroups
@task_group
def task_group_function(value: int) -> None:
"""TaskGroup for grouping related Tasks"""
task_3(task_2.expand(value=task_1(value)))

# Executing Tasks and TaskGroups
with DAG(
dag_id="example_task_group_decorator",
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
tags=["example"],
) as dag:
start_task = task_start()
end_task = task_end()
for i in range(5):
current_task_group = task_group_function(i)
start_task >> current_task_group >> end_task

# [END howto_task_group_decorator]

Operating System

Manjaro Linux

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
I wanted to discuss an important change proposal for our Breeze2 approach.

TL;DR; I thought a lot about how we shold run Breeze2. Instead of having a single ./breeze command we could have multiple airflow-* commands that would be automatically installed by pipx install -e ./dev/breeze and that it should be the only way of running various commands of breeze (both in CI and during local development). An example of that is airflow-freespace command that is used in CI to free space needed to run the commands.

Context

I looked at some of the changes made recently, particularly just merged: #20701 and I thought we might change the approach of the new Breeze following the learnings we had and mostly suggestions of @uranusjr when we dicussed the custom code in https://github.com/apache/airflow/blob/main/Breeze2 for bootstraping the environment.

This is - in essence very similar aas the old ./breeze was and I kept to be "anchored" with it however by introducing pipx and installing ./breeze locally in editable mode, we've almost achieved what I wanted from the beginning - i.e. very simple and os-independent way of bootstrapping the environment for Breeze.

Current state, reasoning and decision is captured here (still Draft though): https://github.com/apache/airflow/blob/main/dev/breeze/doc/adr/0003-bootstraping-virtual-environment.md

Proposal

The "free-space" script shows how we can utilize multiple entry-points,. Instead of "one-command to rule them all" - "Breeze" - we could have multiple commands/entry-points installed with pipx:

airflow-breeze -> to run Breeze shell

airflow-build-ci-image -> to build ci image

airflow-build-prod-image-> to build prod image

airflow-build-docs - > to build docs,

airflow-run-static-checks - > to run static checks

and so on, rather than:

./Breeze2 build-ci-image

./Breeze2 build-docs

and so on.

Why it might be better ?

Mainly because of zen of python: https://www.python.org/dev/peps/pep-0020/

There should be one-- and preferably only one --obvious way to do it.

Wile pipx-only approach is not perfect, I think we can make it close to it (with some effort and added complexity).

With PIPX installation, all those commands are easily accessible in the PATH (and auto-complete'able).

Also the very same commands could be used in CI (same as we did with "airflow-freespace" - this would limit some duplication and made it much more straightforward - how to run some commands locally. The parameters for those commands could still be passed by environment variables in CI (Click has support for that and we already use it).

The only problem to solve is how to handle errors when somene has "old" "breeze" virtualenv. Breeze will change dynamically - new dependencies will be added and we will move to newer versions regularly. This might lead to unnecessary issues and questions on Slack if those errorrs are somehow mysterious and not obvious. Current ./Breeze2 bootstrap script does it by automated upgrading of the virtualenv. However I thought about it and maybe we can approach it differently.

I think we have a way to handle it. We should write a little common code for those commands to capture and store in repo what dependencies are needed (and which versions - that could be even requirements.txt file) and update it automatically with pre-commit and warn the users in cases the dependencies are missing or outdated whenever any command is run. Those dependencies will not change often, but when they do, all users should know that they should upgrade and it should be possible.

This would be a warning only but it could provide a helpful "Your breeze enviroinment is outdated, please run pipx install -e ./dev/breeze --force to update the environment" warning.

Then we could (as @uranusjr proposed and stronly suggested) remove the custom ./Breeze2 bootstrapping.

Why it might be worse ?

There are two serious dangers with pipx:

that the version you have in one branch will not work with another branch (very likely from the past experiences)

that even if you are in the same branch, you could have not installed the latest version of dependencies (and Breeze will fail) or that you have not installed it with --editable flag - which means that the code of Breeze in the branch is different that what you run. This might lead to serious, unforeseen and difficult to diagnose problems.

Both are real dangers, and they will significantly increase the likelihood that different users (especially the ones who are not experienced) will get Breeze not working and that they will raise support questions or (worse) that they will consider Breeze as buggy and will avoid using it or even will drop the idea of contributing to Airflow.

The ./Breeze2 bootstrapping is free of those problems:

--editable mode is automatically used

latest dependencies are added when they were missing automatically

your Breeze version is very tightly connecteect to the branch you are in