Repository: apache/airflow

Issue:
Could not get scheduler_job_id
**Apache Airflow version:**

2.0.0

**Kubernetes version (if you are using kubernetes) (use kubectl version):**

1.18.3

**Environment:**

Cloud provider or hardware configuration: AWS

**What happened:**

When trying to run a DAG, it gets scheduled, but task is never run. When attempting to run task manually, it shows an error:

```
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: airflow-web-ffdd89d6-h98vj
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 1366, in run
    executor.start()
  File "/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
```

**What you expected to happen:**

The task to be run successfully without 

**How to reproduce it:**

Haven't pinpointed what causes the issue, besides an attempted upgrade from Airflow 1.10.14 to Airflow 2.0.0

**Anything else we need to know:**

This error is encountered in an upgrade of Airflow from 1.10.14 to Airflow 2.0.0

EDIT: Formatted to fit the issue template


Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Does it only happen with Kubernetes Executor?
Looks like a bug with Kubernetes Executor. Related issue: https://github.com/apache/airflow/issues/13805

> Does it only happen with Kubernetes Executor?

Yes, it happens only with Kubernetes Executor, apparently. Tried LocalExecutor and task completes without problems when triggering the DAG.
Hi @ClassyLion can you post the logs leading up to that error? Does the executor just never start?

What's strange is that this error seems to suggest that the executor never receives a job_id.
I currently don't have access to the specific AirFlow deployment, but I checked logs thoroughly multiple times and there were no issues with anything starting. This was the only problematic issue that appeared. 
When I get access to the deployment, I'll update with specific information.

Just checking here --- This only happens when you try to run the "task" in the UI. 

Can you try by just triggering the DAG itself and not an individual Task please
If I only trigger the DAG, the task will not run. It will be queued but not finished, or even running. And the DAG stays as running without any progress.
EDIT: That is the reason, why I tried to run the "task" in the UI.
Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.

Did you unpause the DAG too?
Other DAGs don't run aswell. I have logging set to DEBUG and it didn't change anything in the logs. I will share logs tomorrow, when I get access to deployment.
I unpaused the dag and it got scheduled and started, but the tasks were not running, therefore, not finishing.
I am facing the same error while trying to backfill:
Logs:
```
[2021-01-27 06:35:50,209] {airflow-log-cleanup.py:44} INFO - ENABLE_DELETE_CHILD_LOG  False
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:320 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2021-01-27 06:35:50,376] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
    run_backwards=args.run_backwards,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1701, in run
    job.run()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
    executor.start()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
```
> Can you post your DAG please? Does other DAGs run ? Can you post the logs from the Scheduler too with DEBUG level logging.
> 
> Did you unpause the DAG too?

```
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2021-01-27 10:07:10,148] {scheduler_job.py:1241} INFO - Starting the scheduler
[2021-01-27 10:07:10,149] {scheduler_job.py:1246} INFO - Processing each file at most -1 times
[2021-01-27 10:07:10,150] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
[2021-01-27 10:07:10,186] {kubernetes_executor.py:462} INFO - When executor started up, found 0 queued task instances
[2021-01-27 10:07:10,192] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 42
[2021-01-27 10:07:10,193] {scheduler_job.py:1751} INFO - Resetting orphaned tasks for active dag runs
[2021-01-27 10:07:10,193] {kubernetes_executor.py:126} INFO - Event: and now my watch begins starting at resource_version: 0
[2021-01-27 10:07:10,211] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2021-01-27 10:11:15,626] {scheduler_job.py:938} INFO - 2 tasks up for execution:
        <TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
        <TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,635] {scheduler_job.py:967} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 2 task instances ready to be queued
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 0/16 running and queued tasks
[2021-01-27 10:11:15,635] {scheduler_job.py:995} INFO - DAG test.monthly.weekly.trigger has 1/16 running and queued tasks
[2021-01-27 10:11:15,636] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
        <TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-26 16:58:38.208907+00:00 [scheduled]>
        <TaskInstance: test.monthly.weekly.trigger.test_task 2021-01-27 10:11:06.043581+00:00 [scheduled]>
[2021-01-27 10:11:15,639] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,640] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2021-01-27 10:11:15,640] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py']
[2021-01-27 10:11:15,684] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
/usr/local/lib/python3.8/site-packages/airflow/kubernetes/pod_generator.py:193 DeprecationWarning: Using a dictionary for the executor_config is deprecated and will soon be removed.please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key instead.
[2021-01-27 10:11:15,901] {kubernetes_executor.py:510} INFO - Add task TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 27, 10, 11, 6, 43581, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-27T10:11:06.043581+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'] with executor_config {'KubernetesExecutor': {'request_memory': '256Mi', 'limit_memory': '512Mi'}}
[2021-01-27 10:11:15,906] {kubernetes_executor.py:277} INFO - Kubernetes job is (TaskInstanceKey(dag_id='test.monthly.weekly.trigger', task_id='test_task', execution_date=datetime.datetime(2021, 1, 26, 16, 58, 38, 208907, tzinfo=Timezone('UTC')), try_number=1), ['airflow', 'tasks', 'run', 'test.monthly.weekly.trigger', 'test_task', '2021-01-26T16:58:38.208907+00:00', '--local', '--pool', 'default_pool', '--subdir', '/usr/local/airflow/dags/repo/dags/test.monthly.weekly.trigger.py'], {'api_version': 'v1',
 'kind': 'Pod',
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': None,
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': None,
              'managed_fields': None,
              'name': None,
              'namespace': None,
              'owner_references': None,
              'resource_version': None,
              'self_link': None,
              'uid': None},
 'spec': {'active_deadline_seconds': None,
          'affinity': None,
          'automount_service_account_token': None,
          'containers': [{'args': [],
                          'command': [],
                          'env': [],
                          'env_from': [],
                          'image': None,
                          'image_pull_policy': None,
                          'lifecycle': None,
                          'liveness_probe': None,
                          'name': 'base',
                          'ports': [],
                          'readiness_probe': None,
                          'resources': {'limits': {'memory': '512Mi'},
                                        'requests': {'memory': '256Mi'}},
                          'security_context': None,
                          'stdin': None,
                          'stdin_once': None,
                          'termination_message_path': None,
                          'termination_message_policy': None,
                          'tty': None,
                          'volume_devices': None,
                          'volume_mounts': [],
                          'working_dir': None}],
          'dns_config': None,
          'dns_policy': None,
          'enable_service_links': None,
          'host_aliases': None,
          'host_ipc': None,
          'host_network': False,
          'host_pid': None,
          'hostname': None,
          'image_pull_secrets': [],
          'init_containers': None,
          'node_name': None,
          'node_selector': None,
          'preemption_policy': None,
          'priority': None,
          'priority_class_name': None,
          'readiness_gates': None,
          'restart_policy': None,
          'runtime_class_name': None,
          'scheduler_name': None,
          'security_context': None,
          'service_account': None,
          'service_account_name': None,
          'share_process_namespace': None,
          'subdomain': None,
          'termination_grace_period_seconds': None,
          'tolerations': None,
          'volumes': []},
 'status': None}, None)
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-26 16:58:38.208907+00:00 exited with status queued for try_number 1
[2021-01-27 10:11:15,984] {scheduler_job.py:1193} INFO - Executor reports execution of test.monthly.weekly.trigger.test_task execution_date=2021-01-27 10:11:06.043581+00:00 exited with status queued for try_number 1
[2021-01-27 10:11:15,987] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type ADDED
[2021-01-27 10:11:15,988] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending
[2021-01-27 10:11:15,992] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type MODIFIED
[2021-01-27 10:11:15,992] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending
[2021-01-27 10:11:15,996] {kubernetes_executor.py:147} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 had an event of type MODIFIED
[2021-01-27 10:11:15,996] {kubernetes_executor.py:202} INFO - Event: testmonthlyweeklytriggertesttask-b2dda5ecac3c44b9870e38a018c4c421 Pending

```
Seems to be related to this commit: https://github.com/apache/airflow/commit/3ca11eb9b02a2c2591292fd6b76e0e98b8f22656

In `airflow/jobs/scheduler_job.py`, `executor.job_id` is set but not for `airflow/jobs/backfill_job.py`. The same can be observed in `airflow/www/views.py` too.
So the solution here would be to give an arbitrary scheduler_job_id "a.k.a. 'backfill' or 'manual'" For these two use-cases.

@yjwong @ClassyLion would either of you be interested in taking this on? I'd be glad to help you get set up with a dev environment/take you through the PR process (no problem if you don't have the cycles but this seems like a pretty great first ticket :) ).
> So the solution here would be to give an arbitrary scheduler_job_id "a.k.a. 'backfill' or 'manual'" For these two use-cases.
> 
> @yjwong @ClassyLion would either of you be interested in taking this on? I'd be glad to help you get set up with a dev environment/take you through the PR process (no problem if you don't have the cycles but this seems like a pretty great first ticket :) ).

I can say that I am interested and I'm up for it... At least to try it.
@ClassyLion Awesome! Glad to help you get set up :).

So here is the documentation on getting a dev environment set up https://github.com/apache/airflow/blob/master/CONTRIBUTORS_QUICK_START.rst

Please let me know if you run into any roadbumps (would be good for me to know so I can fix them)

After that I'm glad to jump on a zoom to help or feel free to comment here and I can answer any questions :)
Hey, @ClassyLion I don't mean to poach this one and I would be happy to collaborate, but this is holding up our deployment so I thought I would get started. @dimberman PR #14160 adds the hardcoded `job_id`'s, but I could use some guidance on how best to add test coverage. i.e. I'm still getting familiar with the testing landscape and I'm not sure if it would make more sense to add tests to an existing file or if this is new territory.
Hey, @MaxTaggart. I understand how it is. I managed to fully setup the dev environment on my side and made some attempts at hardcoding the values, but it didn't work for me. Also, dayjob takes requires attention as well, resulting in me not being able dedicate as much time as I would want to.
Thank you @MaxTaggart.

Here is a test that should work for the BackFillJob

```
    def test_backfill_has_job_id(self):
        dag = self.dagbag.get_dag("test_start_date_scheduling")
        dag.clear()

        executor = MockExecutor(parallelism=16)

        job = BackfillJob(
            executor=executor,
            dag=dag,
            start_date=DEFAULT_DATE,
            end_date=DEFAULT_DATE + datetime.timedelta(days=1),
            run_backwards=True,
        )
        job.run()
        assert executor.job_id != None
```

and here is one that should work for test_views.py
```
    @mock.patch('airflow.executors.executor_loader.ExecutorLoader.get_default_executor')
    def test_run_executor_has_job_id(self, get_default_executor_function):
        executor = CeleryExecutor()
        executor.heartbeat = lambda: True
        get_default_executor_function.return_value = executor

        task_id = 'runme_0'

        form = dict(
            task_id=task_id,
            dag_id="example_bash_operator",
            ignore_all_deps="false",
            ignore_ti_state="false",
            execution_date=self.EXAMPLE_DAG_DEFAULT_DATE,
            origin='/home',
        )
        resp = self.client.post('run', data=form, follow_redirects=True)
        assert executor.job_id !=None
```

Please let me know if those work as expected.

@ClassyLion Sorry this one didn't work out but now that you're set up if you wanna pick any of the "first issue" tagged issues for the project I'm still glad to help you get a commit in :)
@dimberman Thanks for those tests, they are passing now and I pushed a new commit to that PR that includes them. Just out of curiosity, do we not need to create a unique `job_id` for each backfill/manual job? (Instead of hardcoding a constant)
@MaxTaggart the job_id is mostly to identify which scheduler launched the job. With how Scheduler HA works in 2.0, once the tasks launches, the other schedulers should be able to adopt the orphan task. No unique names needed :)
@MaxTaggart you might need to rebase from master to get tests passing
@dimberman It looks like PR #14160  is getting hung up on a language check that doesn't like "WhiteListRoundRobin" from the cassandra hook in `airflow/providers/apache/cassandra/hooks/cassandra.py`. What's weird is that the file should be excluded from that check since it is in the exclude list in `.pre-commit-config.yaml:317`:

``` yaml
 - id: language-matters
        language: pygrep
        name: Check for language that we do not accept as community
        description: Please use "deny_list" or "allow_list"  instead.
        entry: "(?i)(black|white)[_-]?list"
        pass_filenames: true
        exclude: >
          (?x)
          ^airflow/providers/apache/cassandra/hooks/cassandra.py$|
          ^airflow/providers/apache/hive/operators/hive_stats.py$|
          ^airflow/providers/apache/hive/.*PROVIDER_CHANGES_*|
          ^airflow/providers/apache/hive/.*README.md$|
          ^tests/providers/apache/cassandra/hooks/test_cassandra.py$|
          ^docs/apache-airflow-providers-apache-cassandra/connections/cassandra.rst$|
          ^docs/apache-airflow-providers-apache-hive/commits.rst$|git|
          ^CHANGELOG.txt$
```

Any ideas?
@MaxTaggart It is complaining about the `tags` file you added, not cassandra.py itself. You probably didn't mean to commit the tags file anyway :) 
Aha, good call, I did not. Embarassing...
Alright @dimberman or @ashb, I need to raise the flag again. The CI/CD is now failing in two spots. I can't find any error messages and I tried re-running it after catching up with `master` but it still isn't passing. These are the checks that are failing:

1. MySQL8, Py3.6: Always Core Other API CLI Providers WWW Integration Heisentests
2. MySQL8, Py3.8: Always Core Other API CLI Providers WWW Integration Heisentests

Is it possible that the `BackFillJob` test is missing a mark, or is in the wrong section?
I'll take a look in my morning.
Hey @ashb, have you had a chance to take a look at this? I'm also happy to do more digging if you have any hunches about what might be going wrong.
I also encounter this problem, but curious about another questions, why load same DAG three times before the exception happen, and the last two of loaded DAG path seems incorrect.  
 
```
[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
```
 
```
airflow@webserver-6b975954d-pf6h8:/opt/airflow$ airflow dags backfill etl_dag1 -s 2020-01-20 -e 2020-01-21
[2021-02-19 03:35:48,536] {settings.py:210} DEBUG - Setting up DB connection pool (PID 214)
[2021-02-19 03:35:48,537] {settings.py:281} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=214
[2021-02-19 03:35:48,636] {cli_action_loggers.py:40} DEBUG - Adding <function default_action_log at 0x7f7b77054320> to pre execution callback
[2021-02-19 03:35:50,016] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f7b77054320>]
/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py:62 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
[2021-02-19 03:35:50,050] {dagbag.py:448} INFO - Filling up the DagBag from /home/airflow/dags
[2021-02-19 03:35:50,051] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/code-dag.py
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,240] {executor_loader.py:82} DEBUG - Loading core executor: KubernetesExecutor
[2021-02-19 03:35:50,398] {kubernetes_executor.py:473} INFO - Start Kubernetes executor
[2021-02-19 03:35:50,415] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
    run_backwards=args.run_backwards,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1706, in run
    job.run()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
    executor.start()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 475, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-02-19 03:35:50,429] {settings.py:292} DEBUG - Disposing DB connection pool (PID 214)
```


Is it possible to implement this fix in deployment?
Hi all,
Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill. 
> Hi all,
> Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill.

The fix will be available in Airflow 2.0.2 -- which should be released in a week or so
> > Hi all,
> > Thank you for investigating into this. May I ask, when can we expect to have this fix released? Because it is preventing us from doing a manual backfill.
> 
> The fix will be available in Airflow 2.0.2 -- which should be released in a week or so

Hi @kaxil, Thanks for getting back. In this case we will wait a few more weeks 😉
We're actually running into this issue just running the `CeleryKubernetesExecutor` without doing backfill at all. Same issue as airflow-helm/charts#114.
@dimberman unless I'm mistaken, it appears that although @MaxTaggart fixed the backfill issue related to this issue, the run command is still throwing the original error in 2.0.2.

Would adding `job_id = 'manual'` in airflow/cli/commands/task_command.py fix the issue you think? I noticed it doesn't look it job_id getting set when using the Kubernetes Executor.

```
def _run_task_by_executor(args, dag, ti):
    """
    Sends the task to the executor for execution. This can result in the task being started by another host
    if the executor implementation does
    """
    pickle_id = None
    if args.ship_dag:
        try:
            # Running remotely, so pickling the DAG
            with create_session() as session:
                pickle = DagPickle(dag)
                session.add(pickle)
                pickle_id = pickle.id
                # TODO: This should be written to a log
                print(f'Pickled dag {dag} as pickle_id: {pickle_id}')
        except Exception as e:
            print('Could not pickle the DAG')
            print(e)
            raise e
    executor = ExecutorLoader.get_default_executor()
    executor.job_id = 'manual'
    executor.start()
    print("Sending to executor.")
    executor.queue_task_instance(
        ti,
        mark_success=args.mark_success,
        pickle_id=pickle_id,
        ignore_all_deps=args.ignore_all_dependencies,
        ignore_depends_on_past=args.ignore_depends_on_past,
        ignore_task_deps=args.ignore_dependencies,
        ignore_ti_state=args.force,
        pool=args.pool,
    )
    executor.heartbeat()
    executor.end()
```

If so, wondering how i should test it. Happy to take this on. 

Resolving patch:
diff --git a/airflow/cli/commands/task_command.py b/airflow/cli/commands/task_command.py
--- a/airflow/cli/commands/task_command.py
+++ b/airflow/cli/commands/task_command.py
@@ -88,6 +88,7 @@ def _run_task_by_executor(args, dag, ti):
             print(e)
             raise e
     executor = ExecutorLoader.get_default_executor()
+    executor.job_id = "manual"
     executor.start()
     print("Sending to executor.")
     executor.queue_task_instance(


Test patch: