internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
mitmproxy crashed error after application connects to a websocket
### Discussed in https://github.com/mitmproxy/mitmproxy/discussions/6601

<div type='discussions-op-text'>

<sup>Originally posted by **Botytec** January 14, 2024</sup>
im running mitmproxy in local mode that is proxying only one application, everytime that application connects to a websocket server i get this error

mitmproxy has crashed!
Traceback (most recent call last):
  File "mitmproxy\proxy\server.py", line 380, in server_event
  File "mitmproxy\proxy\layer.py", line 156, in handle_event
  File "mitmproxy\proxy\layer.py", line 269, in handle_event
  File "mitmproxy\proxy\layer.py", line 156, in handle_event
  File "mitmproxy\proxy\tunnel.py", line 98, in _handle_event
  File "mitmproxy\proxy\layers\tls.py", line 485, in event_to_child
  File "mitmproxy\proxy\tunnel.py", line 153, in event_to_child
  File "mitmproxy\proxy\layer.py", line 156, in handle_event
  File "mitmproxy\proxy\tunnel.py", line 98, in _handle_event
  File "mitmproxy\proxy\tunnel.py", line 153, in event_to_child
  File "mitmproxy\proxy\layer.py", line 156, in handle_event
  File "mitmproxy\proxy\layers\http\__init__.py", line 880, in _handle_event
  File "mitmproxy\proxy\layers\http\__init__.py", line 936, in event_to_child
  File "mitmproxy\proxy\layer.py", line 147, in handle_event
  File "mitmproxy\proxy\utils.py", line 26, in _check_event_type
AssertionError: Unexpected event type at HttpStream._handle_event: Expected Start|HttpEvent, got Reply(WebsocketStartHook(flow=<HTTPFlow
</div>


Discussion:

Resolving patch:
diff --git a/mitmproxy/proxy/layers/http/__init__.py b/mitmproxy/proxy/layers/http/__init__.py
--- a/mitmproxy/proxy/layers/http/__init__.py
+++ b/mitmproxy/proxy/layers/http/__init__.py
@@ -528,8 +528,8 @@ def flow_done(self) -> layer.CommandGenerator[None]:
                 yield commands.Log(
                     f"{self.debug}[http] upgrading to {self.child_layer}", DEBUG
                 )
-            yield from self.child_layer.handle_event(events.Start())
             self._handle_event = self.passthrough
+            yield from self.child_layer.handle_event(events.Start())
         else:
             yield DropStream(self.stream_id)
 
@@ -749,8 +749,8 @@ def handle_connect_finish(self):
 
         if 200 <= self.flow.response.status_code < 300:
             self.child_layer = self.child_layer or layer.NextLayer(self.context)
-            yield from self.child_layer.handle_event(events.Start())
             self._handle_event = self.passthrough
+            yield from self.child_layer.handle_event(events.Start())
             yield SendHttp(
                 ResponseHeaders(self.stream_id, self.flow.response, True),
                 self.context.client,


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Fresh installed mitmproxy 10.2.3 gives a warning from cryptography 42.0.x on startup
#### Problem Description
Fresh installed mitmproxy 10.2.3 gives a warning from cryptography 42.0.x on startup:
`.../pipx/venvs/mitmproxy/lib/python3.12/site-packages/mitmproxy/certs.py:111: CryptographyDeprecationWarning: Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.`
`  return self._cert.not_valid_after.replace(tzinfo=datetime.timezone.utc)`
May not be a big problem, but feel anxious.

#### Steps to reproduce the behavior:
1. pipx uninstall mitmproxy
2. pipx install mitmproxy
3. run mitmproxy or mitmdump or mitmweb

#### System Information
Mitmproxy: 10.2.3
Python:    3.12.2
OpenSSL:   OpenSSL 3.2.1 30 Jan 2024
Platform:  macOS-10.13.6-x86_64-i386-64bit

cryptography:  42.0.5



Discussion:

Resolving patch:
diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py
--- a/mitmproxy/certs.py
+++ b/mitmproxy/certs.py
@@ -100,15 +100,12 @@ def issuer(self) -> list[tuple[str, str]]:
 
     @property
     def notbefore(self) -> datetime.datetime:
-        # TODO: Use self._cert.not_valid_before_utc once cryptography 42 hits.
-        # x509.Certificate.not_valid_before is a naive datetime in UTC
-        return self._cert.not_valid_before.replace(tzinfo=datetime.timezone.utc)
+        # type definitions haven't caught up with new API yet.
+        return self._cert.not_valid_before_utc  # type: ignore
 
     @property
     def notafter(self) -> datetime.datetime:
-        # TODO: Use self._cert.not_valid_after_utc once cryptography 42 hits.
-        # x509.Certificate.not_valid_after is a naive datetime in UTC
-        return self._cert.not_valid_after.replace(tzinfo=datetime.timezone.utc)
+        return self._cert.not_valid_after_utc  # type: ignore
 
     def has_expired(self) -> bool:
         if sys.version_info < (3, 11):  # pragma: no cover


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Fix CryptographyDeprecationWarnings
When Cryptography>=42.0, CryptographyDeprecationWarnings will be displayed.

#### Description

<!-- describe your changes here -->

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.



Discussion:

Resolving patch:
diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py
--- a/mitmproxy/certs.py
+++ b/mitmproxy/certs.py
@@ -100,16 +100,23 @@ def issuer(self) -> list[tuple[str, str]]:
 
     @property
     def notbefore(self) -> datetime.datetime:
-        # type definitions haven't caught up with new API yet.
-        return self._cert.not_valid_before_utc  # type: ignore
+        try:
+            # type definitions haven't caught up with new API yet.
+            return self._cert.not_valid_before_utc  # type: ignore
+        except AttributeError:  # pragma: no cover
+            # cryptography < 42.0
+            return self._cert.not_valid_before.replace(tzinfo=datetime.timezone.utc)
 
     @property
     def notafter(self) -> datetime.datetime:
-        return self._cert.not_valid_after_utc  # type: ignore
+        try:
+            return self._cert.not_valid_after_utc  # type: ignore
+        except AttributeError:  # pragma: no cover
+            return self._cert.not_valid_after.replace(tzinfo=datetime.timezone.utc)
 
     def has_expired(self) -> bool:
         if sys.version_info < (3, 11):  # pragma: no cover
-            return datetime.datetime.utcnow() > self._cert.not_valid_after
+            return datetime.datetime.now(datetime.timezone.utc) > self.notafter
         return datetime.datetime.now(datetime.UTC) > self.notafter
 
     @property


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Mitmdump does not exit
#### Problem Description
Mitmdump does not exit automatically when executing:

`mitmdump -nr infile -w outfile
`

Until version 10.0.0 it was working properly and when running mitmdump with "-n" the process finished automatically once the outfile was written.

#### Steps to reproduce the behavior:
1. Generate a mitm file
2. Execute mitmdump -nr infile -w outfile

#### System Information
Mitmproxy: 10.3.0 binary
Python:    3.12.3
OpenSSL:   OpenSSL 3.2.1 30 Jan 2024
Platform:  Linux-6.5.0-27-generic-x86_64-with-glibc2.35



Discussion:
Thanks for the report. Bisected to 6e38a56f4c399551c3fe3d399bddfa8083ccd0f1.

Resolving patch:
diff --git a/mitmproxy/addons/readfile.py b/mitmproxy/addons/readfile.py
--- a/mitmproxy/addons/readfile.py
+++ b/mitmproxy/addons/readfile.py
@@ -71,8 +71,6 @@ async def doread(self, rfile: str) -> None:
             await self.load_flows_from_path(rfile)
         except exceptions.FlowReadException as e:
             logger.exception(f"Failed to read {ctx.options.rfile}: {e}")
-        finally:
-            self._read_task = None
 
     def running(self):
         if ctx.options.rfile:
@@ -80,7 +78,7 @@ def running(self):
 
     @command.command("readfile.reading")
     def reading(self) -> bool:
-        return bool(self._read_task)
+        return bool(self._read_task and not self._read_task.done())
 
 
 class ReadFileStdin(ReadFile):


Test patch:
diff --git a/test/mitmproxy/addons/test_readfile.py b/test/mitmproxy/addons/test_readfile.py
--- a/test/mitmproxy/addons/test_readfile.py
+++ b/test/mitmproxy/addons/test_readfile.py
@@ -54,13 +54,21 @@ async def test_read(self, tmpdir, data, corrupt_data, caplog_async):
 
             tf = tmpdir.join("tfile")
 
-            with mock.patch("mitmproxy.master.Master.load_flow") as mck:
-                tf.write(data.getvalue())
-                tctx.configure(rf, rfile=str(tf), readfile_filter=".*")
-                mck.assert_not_awaited()
-                rf.running()
+            load_called = asyncio.Event()
+
+            async def load_flow(*_, **__):
+                load_called.set()
+
+            tctx.master.load_flow = load_flow
+
+            tf.write(data.getvalue())
+            tctx.configure(rf, rfile=str(tf), readfile_filter=".*")
+            assert not load_called.is_set()
+            rf.running()
+            await load_called.wait()
+
+            while rf.reading():
                 await asyncio.sleep(0)
-                mck.assert_awaited()
 
             tf.write(corrupt_data.getvalue())
             tctx.configure(rf, rfile=str(tf))
Repository: mitmproxy/mitmproxy

Issue:
httpauth is not attached to replay request 
#### Problem Description
I set mitmproxy to run in reverse mode as a proxy to real server, and then protect mitmproxy with a pair of user:pass in the proxyauth option. A regular request would go through, but a reply of that same request would return 401 Unauthorized

#### Steps to reproduce the behavior:
1. turn on reverse mode in mitmweb
2. set basic auth in proxyauth in 'username:pass' format
3. initiate a success request
4. replay the request

#### System Information
Mitmproxy: 10.1.5
Python:    3.11.6
OpenSSL:   OpenSSL 3.1.4 24 Oct 2023
Platform:  Linux-4.14.276-211.499.amzn2.x86_64-x86_64-with-glibc2.31



Discussion:

Resolving patch:
diff --git a/mitmproxy/addons/proxyauth.py b/mitmproxy/addons/proxyauth.py
--- a/mitmproxy/addons/proxyauth.py
+++ b/mitmproxy/addons/proxyauth.py
@@ -76,6 +76,8 @@ def requestheaders(self, f: http.HTTPFlow) -> None:
             # Is this connection authenticated by a previous HTTP CONNECT?
             if f.client_conn in self.authenticated:
                 f.metadata["proxyauth"] = self.authenticated[f.client_conn]
+            elif f.is_replay:
+                pass
             else:
                 self.authenticate_http(f)
 


Test patch:
diff --git a/test/mitmproxy/addons/test_proxyauth.py b/test/mitmproxy/addons/test_proxyauth.py
--- a/test/mitmproxy/addons/test_proxyauth.py
+++ b/test/mitmproxy/addons/test_proxyauth.py
@@ -239,6 +239,11 @@ def test_handlers(self):
             assert not f2.response
             assert f2.metadata["proxyauth"] == ("test", "test")
 
+            f3 = tflow.tflow()
+            f3.is_replay = True
+            up.requestheaders(f3)
+            assert not f2.response
+
 
 @pytest.mark.parametrize(
     "spec",
Repository: vaexio/vaex

Issue:
ImportError: cannot import name 'Task'
https://github.com/vaexio/vaex/blob/f654e895e48a1c5080ad9f84df9a2c31c7add64a/packages/vaex-ui/vaex/ui/layers.py#L2740

Sorry but this version does not find Task in `vaex.dataset`. Should this be imported from tasks instead?


Discussion:

Resolving patch:
diff --git a/packages/vaex-ui/vaex/ui/layers.py b/packages/vaex-ui/vaex/ui/layers.py
--- a/packages/vaex-ui/vaex/ui/layers.py
+++ b/packages/vaex-ui/vaex/ui/layers.py
@@ -2737,5 +2737,6 @@ def update(self):
             self.flag_needs_update()
             self.plot_window.queue_update()
 
-from vaex.dataset import Dataset, Task
+from vaex.dataset import Dataset
+from vaex.tasks import Task
 from vaex.ui.plot_windows import PlotDialog


Test patch:
Repository: vaexio/vaex

Issue:
Pandas dependency
We now depends on Pandas:
https://github.com/vaexio/vaex/blob/255ccbc192d54c619a273de21a05f919da8ffadf/packages/vaex-core/vaex/formatting.py

Introduced in https://github.com/vaexio/vaex/pull/192

We should not depend on pandas, it is not a dependency of vaex-core and should not become, we might also grow to large to run on AWS Lambda.


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/formatting.py b/packages/vaex-core/vaex/formatting.py
--- a/packages/vaex-core/vaex/formatting.py
+++ b/packages/vaex-core/vaex/formatting.py
@@ -1,7 +1,7 @@
 import numpy as np
 import numbers
 import six
-import pandas as pd
+import datetime
 
 
 MAX_LENGTH = 50
@@ -15,9 +15,24 @@ def _format_value(value):
     elif isinstance(value, np.ma.core.MaskedConstant):
         value = str(value)
     if isinstance(value, np.datetime64):
-        value = str(pd.to_datetime(value))
+        if np.isnat(value):
+            value = 'NaT'
+        else:
+            value = ' '.join(str(value).split('T'))
     if isinstance(value, np.timedelta64):
-        value = str(pd.to_timedelta(value))
+        if np.isnat(value):
+            value = 'NaT'
+        else:
+            tmp = datetime.timedelta(seconds=value / np.timedelta64(1, 's'))
+            ms = tmp.microseconds
+            s = np.mod(tmp.seconds, 60)
+            m = np.mod(tmp.seconds//60, 60)
+            h = tmp.seconds // 3600
+            d = tmp.days
+            if ms:
+                value = str('%i days %+02i:%02i:%02i.%i' % (d,h,m,s,ms))
+            else:
+                value = str('%i days %+02i:%02i:%02i' % (d,h,m,s))
     elif not isinstance(value, numbers.Number):
         value = str(value)
     if isinstance(value, float):


Test patch:
Repository: vaexio/vaex

Issue:
Deletion of a (virtual) column shows up in autocomplete (jupyter, ipython)
I am not 100% sure if this is the correct way we want to "delete" in-memory columns, in this example:
```
import vaex
ds = vaex.example()
del ds['x']
```
in Jupyter and IPython one still gets `ds.x` to show up via the auto-complete, while of course, the "x" column is no longer accessible.

(Note to anyone reading this: `del ds['x']` does not modify the original hdf5/arrow/csv file on disk. Well not unless one exports the `ds` and overwrites the original file. )


Discussion:
I wouldn't say in memory, any column. I'll see if I can fix it.

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -4293,6 +4293,12 @@ def __delitem__(self, item):
             self.column_names.remove(name)
         else:
             raise KeyError('no such column or virtual_columns named %r' % name)
+        if hasattr(self, name):
+            try:
+                if isinstance(getattr(self, name), Expression):
+                    delattr(self, name)
+            except:
+                pass
 
     @docsubst
     def drop(self, columns, inplace=False, check=True):


Test patch:
diff --git a/tests/drop_test.py b/tests/drop_test.py
--- a/tests/drop_test.py
+++ b/tests/drop_test.py
@@ -29,3 +29,20 @@ def test_drop_depending_filtered(ds_filtered):
     assert 'x' not in ds.get_column_names()
     assert '__x' in ds.get_column_names(hidden=True)
     ds.y.values  # make sure we can evaluate y
+
+def test_drop_autocomplete(ds_local):
+    ds = ds_local
+    ds['drop'] = ds.m
+    ds['columns'] = ds.m
+
+    del ds['x']
+    assert not hasattr(ds, 'x')
+
+    ds.drop('y', inplace=True)
+    assert not hasattr(ds, 'y')
+
+    # make sure we can also remove with name issues
+    del ds['drop']
+    assert callable(ds.drop)
+    del ds['columns']
+    assert 'm' in ds.columns
Repository: vaexio/vaex

Issue:
Error when exporting df with a column containing only empty strings.
There is an error appearing when one tries to export a DataFrame that contains a column which has only empty strings.

`df.export_arrow` does not seem affected by this, but opening such a file causes an error. 


Discussion:
Thanks!

Resolving patch:
diff --git a/packages/vaex-hdf5/vaex/hdf5/dataset.py b/packages/vaex-hdf5/vaex/hdf5/dataset.py
--- a/packages/vaex-hdf5/vaex/hdf5/dataset.py
+++ b/packages/vaex-hdf5/vaex/hdf5/dataset.py
@@ -224,9 +224,10 @@ def _load_variables(self, h5variables):
 
     def _map_hdf5_array(self, data, mask=None):
         offset = data.id.get_offset()
+        if len(data) == 0 and offset is None:
+            offset = 0 # we don't care about the offset for empty arrays
         if offset is None:  # non contiguous array, chunked arrays etc
             # we don't support masked in this case
-            raise "try"
             column = ColumnNumpyLike(data)
             return column
         else:
@@ -239,7 +240,7 @@ def _map_hdf5_array(self, data, mask=None):
                     if dtype == 'utf32':
                         dtype = np.dtype('U' + str(data.attrs['dlength']))
             #self.addColumn(column_name, offset, len(data), dtype=dtype)
-            array = self._map_array(data.id.get_offset(), dtype=dtype, length=len(data))
+            array = self._map_array(offset, dtype=dtype, length=len(data))
             if mask is not None:
                 mask_array = self._map_hdf5_array(mask)
                 return np.ma.array(array, mask=mask_array, shrink=False)
diff --git a/packages/vaex-hdf5/vaex/hdf5/export.py b/packages/vaex-hdf5/vaex/hdf5/export.py
--- a/packages/vaex-hdf5/vaex/hdf5/export.py
+++ b/packages/vaex-hdf5/vaex/hdf5/export.py
@@ -171,7 +171,8 @@ def export_hdf5(dataset, path, column_names=None, byteorder="=", shuffle=False,
                 indices_shape = (N+1, )
 
                 array = h5column_output.require_dataset('data', shape=data_shape, dtype='S1')
-                array[0] = array[0]  # make sure the array really exists
+                if byte_length > 0:
+                    array[0] = array[0]  # make sure the array really exists
 
                 index_array = h5column_output.require_dataset('indices', shape=indices_shape, dtype=dtype_indices)
                 index_array[0] = index_array[0]  # make sure the array really exists


Test patch:
diff --git a/tests/export_test.py b/tests/export_test.py
--- a/tests/export_test.py
+++ b/tests/export_test.py
@@ -2,6 +2,16 @@
 import os
 import tempfile
 
+
+@pytest.mark.parametrize("filename", ["test.hdf5", "test.arrow"])
+def test_export_empty_string(tmpdir, filename):
+	path = str(tmpdir.join('test.hdf5'))
+	s = np.array(["", ""])
+	df = vaex.from_arrays(s=s)
+	df.export(path)
+	vaex.open(path)
+
+
 def test_export(ds_local, tmpdir):
 	ds = ds_local
 	# TODO: we eventually want to support dtype=object, but not for hdf5
Repository: vaexio/vaex

Issue:
Simple issue with DataFrame slicing out-of-bounds case
We have an issue when an out-of-bounds index is specified when slicing a DataFrame. Consider this example:

```
import vaex
df = vaex.from_arrays(x = [0, 1, 2, 3, 4, 5, 6, 7])
```

now 
```
df[:3]
  #    x
  0    0
  1    1
  2    2
```

but when `df[:10]` a `ValueError: array is of length 8, while the length of the DataFrame is 10` is raised. 

I would expect the default behaviour to be that the whole DataFrame is printed. 



Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -4423,6 +4423,9 @@ def __getitem__(self, item):
             start, stop, step = item.start, item.stop, item.step
             start = start or 0
             stop = stop or len(self)
+            if stop < 0:
+                stop = len(self)+stop
+            stop = min(stop, len(self))
             assert step in [None, 1]
             if self.filtered and start == 0:
                 mask = self._selection_masks[FILTER_SELECTION_NAME]


Test patch:
diff --git a/tests/slice_test.py b/tests/slice_test.py
--- a/tests/slice_test.py
+++ b/tests/slice_test.py
@@ -36,3 +36,15 @@ def test_head_with_selection():
     df = vaex.example()
     df.select(df.x > 0, name='test')
     df.head()
+
+
+def test_slice_beyond_end(df):
+    df2 = df[:100]
+    assert df2.x.tolist() == df.x.tolist()
+    assert len(df2) == len(df)
+
+
+def test_slice_negative(df):
+    df2 = df[:-1]
+    assert df2.x.tolist() == df.x.values[:-1].tolist()
+    assert len(df2) == len(df)-1
Repository: vaexio/vaex

Issue:
ds.join with how='right' automatically sets inplace=True
For how='right', inplace is apparently automatically activated.
Using inplace=False does not alter this anymore.
I guess this is a small bug.

ds1.join(ds2, left_on='a', right_on='b', how='right', inplace=False)
returns:

\# | b | y | a | x
-- | -- | -- | -- | --
0 | 'a' | 1 | 'a' | 1
1 | 'b' | 4 | 'b' | 2
2 | 'd' | 9 | masked | masked

For:
ds1

\# | a | x
-- | -- | --
0 | 'a' | 1
1 | 'b' | 2
2 | 'c' | 3

and 
ds2

\# | b | y
-- | -- | --
0 | 'a' | 1
1 | 'b' | 4
2 | 'd' | 9




Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -4957,23 +4957,21 @@ def join(self, other, on=None, left_on=None, right_on=None, lprefix='', rprefix=
         :param inplace: {inplace}
         :return:
         """
-        ds = self if inplace else self.copy()
         inner = False
+        left = self
+        right = other
         if how == 'left':
-            left = ds
-            right = other
+            pass
         elif how == 'right':
-            left = other
-            right = ds
+            left, right = right, left
             lprefix, rprefix = rprefix, lprefix
             lsuffix, rsuffix = rsuffix, lsuffix
             left_on, right_on = right_on, left_on
         elif how == 'inner':
-            left = ds
-            right = other
             inner = True
         else:
             raise ValueError('join type not supported: {}, only left and right'.format(how))
+        left = left if inplace else left.copy()
 
         for name in right:
             if name in left and rprefix + name + rsuffix == lprefix + name + lsuffix:
@@ -4997,7 +4995,7 @@ def join(self, other, on=None, left_on=None, right_on=None, lprefix='', rprefix=
                 else:
                     left.add_column(right_name, right.columns[name])
         else:
-            df = ds
+            df = left
             # we index the right side, this assumes right is smaller in size
             index = right._index(right_on)
             lookup = np.zeros(left._length_original, dtype=np.int64)


Test patch:
diff --git a/tests/join_test.py b/tests/join_test.py
--- a/tests/join_test.py
+++ b/tests/join_test.py
@@ -94,6 +94,7 @@ def test_right_x_x():
     assert df.evaluate('x_r').tolist() == [2, 1, 0]
     assert df.evaluate('y').tolist() == [2, None, 0]
     assert df.evaluate('y_r').tolist() == [None, 1, 2]
+    assert 'y_r' not in df_b
 
 
 def test_left_dup():