internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Accessing merge indicator column causes KeyError with shuffle.method = p2p
**Describe the issue**:
After executing a merge with indicator=True, accessing the merge indicator '_merge' column raises a KeyError. 
Changing the dataframe.shuffle.method to 'disk' instead of 'p2p' resolved the issue.

**Minimal Complete Verifiable Example**:
```python
import pandas as pd
from dask import distributed, dataframe, config
simple_raw = {'id': ['101-a', '102-a', '103-a'],
              'test': ['val101a', 'val102a', 'val103a']}
df = pd.DataFrame(simple_raw, columns=['id',  'test'])

client = distributed.Client(n_workers=2, threads_per_worker=2, memory_limit="2GB")
# config.set({'dataframe.shuffle.method':'disk'})
ddf = dataframe.from_pandas(df, npartitions=2)
merge_results_df = ddf.merge(ddf, on="id", how='outer', indicator=True)
merge_results_df['_merge'].head()
```

**Anything else we need to know?**:
The _merge column is present before calling head()
i.e. `'_merge' in merge_results_df.columns` returns True
After calling head and catching the KeyException,
`'_merge' in merge_results_df.columns` returns False
**Environment**:

- Dask version:  2023.5.0
- Python version: 3.8.15
- Operating System: Ubuntu 20.04.5 LTS
- Install method (conda, pip, source): pip



Discussion:

Resolving patch:
diff --git a/distributed/shuffle/_merge.py b/distributed/shuffle/_merge.py
--- a/distributed/shuffle/_merge.py
+++ b/distributed/shuffle/_merge.py
@@ -172,6 +172,7 @@ def merge_unpack(
     suffixes: Suffixes,
     left_index: bool,
     right_index: bool,
+    indicator: bool = False,
 ):
     from dask.dataframe.multi import merge_chunk
 
@@ -193,6 +194,7 @@ def merge_unpack(
         suffixes=suffixes,
         left_index=left_index,
         right_index=right_index,
+        indicator=indicator,
     )
 
 
@@ -430,5 +432,6 @@ def _construct_graph(self) -> dict[tuple | str, tuple]:
                 self.suffixes,
                 self.left_index,
                 self.right_index,
+                self.indicator,
             )
         return dsk


Test patch:
diff --git a/distributed/shuffle/tests/test_merge.py b/distributed/shuffle/tests/test_merge.py
--- a/distributed/shuffle/tests/test_merge.py
+++ b/distributed/shuffle/tests/test_merge.py
@@ -539,3 +539,22 @@ async def test_merge_does_not_deadlock_if_worker_joins(c, s, a):
         result = await result
     expected = pd.merge(pdf1, pdf2, left_on="a", right_on="x")
     assert_eq(result, expected, check_index=False)
+
+
+@gen_cluster(client=True)
+async def test_merge_indicator(c, s, a, b):
+    data = {
+        "id": [1, 2, 3],
+        "test": [4, 5, 6],
+    }
+    pdf = pd.DataFrame(data)
+    df = dd.from_pandas(pdf, npartitions=2)
+    result = df.merge(df, on="id", how="outer", indicator=True)
+    x = c.compute(result)
+    x = await x
+    expected = pdf.merge(pdf, on="id", how="outer", indicator=True)
+
+    pd.testing.assert_frame_equal(
+        x.sort_values("id", ignore_index=True),
+        expected.sort_values("id", ignore_index=True),
+    )
Repository: dask/distributed

Issue:
CI failing with Bokeh deprecation warnings
> FAILED distributed/dashboard/tests/test_scheduler_bokeh.py::test_events - bokeh.util.warnings.BokehDeprecationWarning: 'circle() method with size value' was deprecated in Bokeh 3.4.0 and will be removed, use 'scatter(size=...) instead' instead.

Example run:
https://github.com/dask/distributed/actions/runs/8338285775/job/22818348113#step:20:5239


Discussion:

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -1922,7 +1922,7 @@ def __init__(self, scheduler, **kwargs):
             **kwargs,
         )
 
-        self.root.circle(
+        self.root.scatter(
             source=self.source,
             x="time",
             y="level",
@@ -2016,7 +2016,7 @@ def __init__(self, scheduler, name, height=150, **kwargs):
             **kwargs,
         )
 
-        self.root.circle(
+        self.root.scatter(
             source=self.source,
             x="time",
             y="y",
@@ -2302,7 +2302,7 @@ def __init__(self, scheduler, **kwargs):
             color="black",
             alpha=0.3,
         )
-        rect = self.root.square(
+        rect = self.root.scatter(
             x="x",
             y="y",
             size=10,
@@ -2310,6 +2310,7 @@ def __init__(self, scheduler, **kwargs):
             source=self.node_source,
             view=node_view,
             legend_field="state",
+            marker="square",
         )
         self.root.xgrid.grid_line_color = None
         self.root.ygrid.grid_line_color = None
@@ -4198,7 +4199,7 @@ def __init__(self, scheduler, width=800, **kwargs):
             min_border_right=0,
             **kwargs,
         )
-        mem_plot.circle(
+        mem_plot.scatter(
             source=self.source, x="memory_percent", y=0, size=10, fill_alpha=0.5
         )
         mem_plot.ygrid.visible = False
@@ -4228,7 +4229,7 @@ def __init__(self, scheduler, width=800, **kwargs):
             min_border_right=0,
             **kwargs,
         )
-        cpu_plot.circle(
+        cpu_plot.scatter(
             source=self.source, x="cpu_fraction", y=0, size=10, fill_alpha=0.5
         )
         cpu_plot.ygrid.visible = False


Test patch:
Repository: dask/distributed

Issue:
`Print host info` CI step raises `RuntimeError: 'pmgr' entry was not found in AppleARMIODevice service` for `Mac OS`
Traceback:

```python
Traceback (most recent call last):
Number of CPUs: 3
  File "/Users/runner/work/distributed/distributed/continuous_integration/scripts/host_info.py", line [5](https://github.com/dask/distributed/actions/runs/8945035724/job/24573207121#step:19:5)3, in <module>
Crude CPU benchmark (higher is better): 69.5
    main()
  File "/Users/runner/work/distributed/distributed/continuous_integration/scripts/host_info.py", line 33, in main
    freqs = psutil.cpu_freq(percpu=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.12/site-packages/psutil/__init__.py", line 1913, in cpu_freq
    ret = _psplatform.cpu_freq()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.12/site-packages/psutil/_psosx.py", line 1[8](https://github.com/dask/distributed/actions/runs/8945035724/job/24573207121#step:19:8)2, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
                       ^^^^^^^^^^^^^^^
RuntimeError: 'pmgr' entry was not found in AppleARMIODevice service
Error: Process completed with exit code 1.
```

See CI logs: https://github.com/dask/distributed/actions/runs/8945035724/job/24573207121#step:19:27


Discussion:
https://github.com/dask/distributed/pull/8628 appears to be the first occurrence of this.
Upstream issue: https://github.com/giampaolo/psutil/issues/2382
> #8628 appears to be the first occurrence of this.

Correlation, not causation: https://github.com/dask/distributed/actions/runs/8972110042/job/24639361479?pr=8637

Resolving patch:
diff --git a/continuous_integration/scripts/host_info.py b/continuous_integration/scripts/host_info.py
--- a/continuous_integration/scripts/host_info.py
+++ b/continuous_integration/scripts/host_info.py
@@ -30,11 +30,16 @@ def main() -> None:
     bench()
     print(f"Crude CPU benchmark (higher is better): {bench():.1f}")
 
-    freqs = psutil.cpu_freq(percpu=True)
-    print("CPU frequency:")
-    for freq in freqs:
-        # FIXME types-psutil
-        print(f"  - current={freq.current}, min={freq.min}, max={freq.max}")  # type: ignore
+    try:
+        freqs = psutil.cpu_freq(percpu=True)
+    # https://github.com/giampaolo/psutil/issues/2382
+    except RuntimeError:
+        print("CPU frequency: not available")
+    else:
+        print("CPU frequency:")
+        for freq in freqs:
+            # FIXME types-psutil
+            print(f"  - current={freq.current}, min={freq.min}, max={freq.max}")  # type: ignore
 
     mem = psutil.virtual_memory()
     print("Memory:")


Test patch:
Repository: dask/distributed

Issue:
distributed hangs on Windows with Python 3.13
**Describe the issue**:

**Minimal Complete Verifiable Example**:

```python
from dask.distributed import Client, Worker, Scheduler, wait
import pyperf

def inc(x):
    return x + 1

async def benchmark():
    async with Scheduler() as scheduler:
        async with Worker(scheduler.address):
            async with Client(scheduler.address, asynchronous=True) as client:
                futures = client.map(inc, range(100))
                for _ in range(10):
                    futures = client.map(inc, futures)
                await wait(futures)

if __name__ == "__main__":
    runner = pyperf.Runner()
    runner.metadata['description'] = "Benchmark dask"
    runner.bench_async_func('dask', benchmark)
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2024.5.0 
- Python version: 3.13 (main branch, future 3.13 beta 1)
- Operating System: Windows 11
- Install method (conda, pip, source): pip


Discussion:

Resolving patch:
diff --git a/distributed/metrics.py b/distributed/metrics.py
--- a/distributed/metrics.py
+++ b/distributed/metrics.py
@@ -2,6 +2,7 @@
 
 import asyncio
 import collections
+import sys
 import threading
 import time as timemod
 from collections.abc import Callable, Hashable, Iterator
@@ -93,7 +94,7 @@ def resync(self) -> None:
 
 
 # A high-resolution wall clock timer measuring the seconds since Unix epoch
-if WINDOWS:
+if WINDOWS and sys.version_info < (3, 13):
     time = _WindowsTime(timemod.time, is_monotonic=False).time
     monotonic = _WindowsTime(timemod.monotonic, is_monotonic=True).time
 else:


Test patch:
Repository: fal-ai/dbt-fal

Issue:
Make logging of insert statements for write_to_source optional or just remove it
`cesarsantos#3303` in Discord asked for this probably to avoid logging sensible information or filling the logger with all the data there is.


Discussion:
The current implementation logs all queries sent to the warehouse in https://github.com/fal-ai/fal/blob/3e10e0f1e635b61be60eaafa3c5c7f5bfabbbc1a/src/faldbt/lib.py#L111
Maybe we could change that to `debug` level?

Resolving patch:
diff --git a/src/faldbt/lib.py b/src/faldbt/lib.py
--- a/src/faldbt/lib.py
+++ b/src/faldbt/lib.py
@@ -108,7 +108,7 @@ def _execute_sql(
     node = _get_operation_node(manifest, project_path, profiles_dir, sql)
     adapter = _get_adapter(project_path, profiles_dir)
 
-    logger.info("Running query\n{}", sql)
+    logger.debug("Running query\n{}", sql)
 
     result = None
     with adapter.connection_for(node):


Test patch:
Repository: fal-ai/dbt-fal

Issue:
AttributeError: 'ParsedSingularTestNode' object has no attribute 'test_metadata'
Since the latest release our fal task has started failing to run a python scripts and gives the following error:

```
Traceback (most recent call last):
  File "/usr/local/bin/fal", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/fal/cli.py", line 146, in cli
    run_fal(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/fal/cli.py", line 129, in run_fal
    _run(
  File "/usr/local/lib/python3.9/site-packages/fal/cli.py", line 185, in _run
    faldbt = FalDbt(
  File "/usr/local/lib/python3.9/site-packages/faldbt/project.py", line 218, in __init__
    self.models, self.tests = _map_nodes_to_models(self._run_results, self._manifest)
  File "/usr/local/lib/python3.9/site-packages/faldbt/project.py", line 519, in _map_nodes_to_models
    tests = manifest.get_tests()
  File "/usr/local/lib/python3.9/site-packages/faldbt/project.py", line 128, in get_tests
    return list(
  File "/usr/local/lib/python3.9/site-packages/faldbt/project.py", line 132, in <lambda>
    lambda node: DbtTest(node=node), self.nativeManifest.nodes.values()
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.9/site-packages/faldbt/project.py", line 54, in __post_init__
    self.name = node.test_metadata.name
AttributeError: 'ParsedSingularTestNode' object has no attribute 'test_metadata'
```


Discussion:
Which version of dbt are you using?
@mkam039 I have tested this in v0.21.1 and v.1.0.0

Resolving patch:
diff --git a/src/faldbt/project.py b/src/faldbt/project.py
--- a/src/faldbt/project.py
+++ b/src/faldbt/project.py
@@ -51,13 +51,16 @@ def __post_init__(self):
         node = self.node
         self.unique_id = node.unique_id
         if node.resource_type == NodeType.Test:
-            self.name = node.test_metadata.name
-
-            # node.test_metadata.kwargs looks like this:
-            # kwargs={'column_name': 'y', 'model': "{{ get_where_subquery(ref('boston')) }}"}
-            # and we want to get 'boston' is the model name that we want extract
-            self.model = re.findall(r"'([^']+)'", node.test_metadata.kwargs['model'])[0]
-            self.column = node.test_metadata.kwargs['column_name']
+            if hasattr(node, "test_metadata"):
+                self.name = node.test_metadata.name
+
+                # node.test_metadata.kwargs looks like this:
+                # kwargs={'column_name': 'y', 'model': "{{ get_where_subquery(ref('boston')) }}"}
+                # and we want to get 'boston' is the model name that we want extract
+                self.model = re.findall(r"'([^']+)'", node.test_metadata.kwargs['model'])[0]
+                self.column = node.test_metadata.kwargs['column_name']
+            else:
+                logger.debug(f"Non-generic test was not processed: {node.name}")
 
     def set_status(self, status: str):
         self.status = status


Test patch:
Repository: fal-ai/dbt-fal

Issue:
Make fal scripts __name__ == '__main__'
**Context**
When users test scripts they will normally have a 

```py
if __name__ == '__main__':
  main()
```

and the `main()` function will be where the whole script lives.

When a script is associated with a model, it is executed "directly". So it could be considered the "__main__" script.

**Describe alternatives you've considered**
Going for the dbt interface of offering a function.

```py
# for models
def model():
  pass
```

```py
# for hooks/after/before scripts
def hook():
  pass



Discussion:
bump! we also ran into this when deploying to prod

Resolving patch:
diff --git a/src/fal/fal_script.py b/src/fal/fal_script.py
--- a/src/fal/fal_script.py
+++ b/src/fal/fal_script.py
@@ -88,6 +88,7 @@ def exec(self, faldbt: FalDbt):
             program = compile(source_code, self.path, "exec")
 
             exec_globals = {
+                "__name__": "__main__",
                 "context": self._build_script_context(),
                 "ref": faldbt.ref,
                 "source": faldbt.source,


Test patch:
diff --git a/integration_tests/features/main_module.feature b/integration_tests/features/main_module.feature
new file mode 100644
--- /dev/null
+++ b/integration_tests/features/main_module.feature
@@ -0,0 +1,13 @@
+Feature: __name__ should be __main__
+  Background: Project Setup
+    Given the project 002_jaffle_shop
+
+  Scenario: main check should be present
+    When the following command is invoked:
+      """
+      fal flow run --profiles-dir $profilesDir --project-dir $baseDir --select +middle_2+
+      """
+    Then the following scripts are ran:
+      | middle_2.middle_script.py |
+    And the script middle_2.middle_script.py output file has the lines:
+      | top name: __main__ | inner name: main_check_2 | passed main if |
diff --git a/integration_tests/projects/002_jaffle_shop/fal_scripts/main_check_2.py b/integration_tests/projects/002_jaffle_shop/fal_scripts/main_check_2.py
new file mode 100644
--- /dev/null
+++ b/integration_tests/projects/002_jaffle_shop/fal_scripts/main_check_2.py
@@ -0,0 +1,3 @@
+def main_check(output):
+    output += f"inner name: {__name__}\n"
+    return output
diff --git a/integration_tests/projects/002_jaffle_shop/fal_scripts/middle_script.py b/integration_tests/projects/002_jaffle_shop/fal_scripts/middle_script.py
--- a/integration_tests/projects/002_jaffle_shop/fal_scripts/middle_script.py
+++ b/integration_tests/projects/002_jaffle_shop/fal_scripts/middle_script.py
@@ -1,10 +1,16 @@
 import os
 from functools import reduce
+from main_check_2 import main_check
 
 model_name = context.current_model.name
 
 output = f"Model name: {model_name}"
-output = output + f"\nStatus: {context.current_model.status}"
+output = output + f"\nStatus: {context.current_model.status}\n"
+
+output += f"top name: {__name__}\n"
+output = main_check(output)
+if __name__ == "__main__":
+    output += "passed main if\n"
 
 temp_dir = os.environ["temp_dir"]
 print(temp_dir)
Repository: fal-ai/dbt-fal

Issue:
Fal read only dbt *.yml files and ignore *.yaml 
<!-- *** Make sure you have searched for an existing bug report for this issue *** -->

**Describe the bug**
In dbt models you can have schems in either .yml or yaml extentions 
also see : https://www.w3schools.io/file/yaml-vs-yml/

**Your environment**
- OS: MacOS
- Paste the following commands output:
```
dbt-fal==1.3.7
fal==0.7.3
dbt-core==1.3.1
```
- Adapter being used: <!-- (if more than one dbt plugin installed) --> 

**How to reproduce**
If you create schema.yaml with fal section the schema will be ignored

**Expected behavior**
schema.yaml should be detected and parsed correctly

**Actual behavior**
Tell us what actually happened.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/src/faldbt/parse.py b/src/faldbt/parse.py
--- a/src/faldbt/parse.py
+++ b/src/faldbt/parse.py
@@ -194,7 +194,10 @@ def get_scripts_list(scripts_dir: str) -> List[str]:
 def get_global_script_configs(source_dirs: List[Path]) -> Dict[str, List[str]]:
     global_scripts = {"before": [], "after": []}
     for source_dir in source_dirs:
+        # Scan directories for .yml files
         schema_files = glob.glob(os.path.join(source_dir, "**.yml"), recursive=True)
+        # Scan directories for .yaml files
+        schema_files += glob.glob(os.path.join(source_dir, "**.yaml"), recursive=True)
         for file in schema_files:
             schema_yml = load_yaml(file)
             if schema_yml is not None:


Test patch:
Repository: fal-ai/dbt-fal

Issue:
Context test column name is None
<!-- *** Make sure you have searched for an existing bug report for this issue *** -->

**Describe the bug**
When accessing `context.current_model.tests` the column name is None if the tests are defined on table level instead of column level. For example these two tests below are doing the same thing, but only the one in `columns` key will have the column name metadata populated in `context`.

```
tests:
  - unique:
      column_name: col
columns:
  - name: col
    tests:
      - unique
```

**Your environment**
- OS: macOS Ventura 13.2.1
- Paste the following commands output:
```
fal --version
fal 0.8.2
dbt --version
Core:
  - installed: 1.4.5

```
- Adapter being used: athena: 1.4.1

**How to reproduce**
1. Define a test in a yml file for an existing dbt model on a table level and specify a column. For example:
```
models:
  - name: model
    tests:
      - unique:
          column_name: col
```
2.  run a python script accessing context.model.tests.column and it should be None even though in the yml file it's defined for column `col`.

**Expected behavior**
I would expect to see `column='col'`

**Actual behavior**
`column=None`

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here.



Discussion:
Thanks for raising this, @hilcik, I'm looking into this.

Resolving patch:
diff --git a/projects/fal/src/faldbt/project.py b/projects/fal/src/faldbt/project.py
--- a/projects/fal/src/faldbt/project.py
+++ b/projects/fal/src/faldbt/project.py
@@ -132,6 +132,10 @@ def __post_init__(self):
         assert isinstance(self.node, GenericTestNode)
         self.column = self.node.column_name
 
+        # Column name might be stored in test_metadata
+        if not self.column and self.node.test_metadata.kwargs.get("column_name"):
+            self.column = self.node.test_metadata.kwargs.get("column_name")
+
     @property
     def source_id(self):
         if self.source_ids:


Test patch:
Repository: systemd/mkosi

Issue:
PostInstallationScript incompatible with Output Format gpt_squashfs (or gpt_ext4 with --minimize)
Environment information:
- mkosi built from git revision: 62fe176d276d3fa0782c283edba5bbb7c5c610c0 (current latest master)
- host: Ubuntu 20.04 (Focal) Server

Reduced `mkosi.default`:
```
[Distribution]
Distribution=ubuntu
Release=groovy

[Output]
Format=gpt_squashfs
Bootable=yes
BootProtocols=uefi

[Packages]
PostInstallationScript=postinst.sh
```

When running `mkosi` to build, the relevant output at the end says:
```
...
‣ Installing Ubuntu complete.
‣ Assigning hostname...
‣ Assigning hostname complete.
‣ Installing boot loader...
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/efi/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/efi/EFI/BOOT/BOOTX64.EFI".
Random seed file /efi/loader/random-seed successfully written (512 bytes).
‣ Installing boot loader complete.
‣ Setting root password...
‣ Setting root password complete.
‣ Running postinstall script...
Failed to stat /dev/loop1p2: No such file or directory
‣ Error: Workspace command /root/postinst final returned non-zero exit code 1.
‣ Unmounting Package Cache...
‣ Unmounting Package Cache complete.
‣ Unmounting image...
‣ Unmounting image complete.
‣ Detaching image file...
‣ Detaching image file complete.
```

Similarly, switching `Format=gpt_squashfs` to `Format=gpt_ext4` and running `mkosi --minimize` produces an identical error output.

Removing `PostInstallationScript=postinst.sh` from `mkosi.default` produces an output image in both these squashfs and minimized ext4 cases, but lacking the ability to run a script at the end of the installation process within the in-progress system.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -2865,7 +2865,8 @@ def nspawn_params_for_blockdev_access(args: CommandLineArguments, loopdev: str)
     for partno in (args.esp_partno, args.bios_partno, args.root_partno, args.xbootldr_partno):
         if partno is not None:
             p = partition(loopdev, partno)
-            params += [f"--bind-ro={p}", f"--property=DeviceAllow={p}"]
+            if os.path.exists(p):
+                params += [f"--bind-ro={p}", f"--property=DeviceAllow={p}"]
     return params
 
 


Test patch: