internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: lisa-lab/pylearn2

Issue:
AttributeError: 'NoneType' object has no attribute 'continue_learning'
If no algorithm is set from the .yaml file (it is the case in kmeans, svm, and others), the training code will always fail with this error:

```
AttributeError: 'NoneType' object has no attribute 'continue_learning'
```

This is cause by [this code in main_loop at pylab2/train.py](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/train.py#L147):

```
        if self.algorithm is None:
          while True:
            if self.exceeded_time_budget(t0, time_budget):
                break
            continue_learning = \
                self.algorithm.continue_learning(self.model)
```



Discussion:
This one is my fault -- that code isn't tested so I didn't notice the error when working on #1090.  I will have a fix up shortly. 


Resolving patch:
diff --git a/pylearn2/train.py b/pylearn2/train.py
--- a/pylearn2/train.py
+++ b/pylearn2/train.py
@@ -143,8 +143,7 @@ def main_loop(self, time_budget=None):
             while True:
                 if self.exceeded_time_budget(t0, time_budget):
                     break
-                continue_learning = \
-                    self.algorithm.continue_learning(self.model)
+                continue_learning = self.model.continue_learning()
                 extension_continue = self.run_callbacks_and_monitoring()
                 assert continue_learning in [True, False, 0, 1]
                 if not continue_learning or not extension_continue:


Test patch:
diff --git a/pylearn2/models/tests/test_kmeans.py b/pylearn2/models/tests/test_kmeans.py
new file mode 100644
--- /dev/null
+++ b/pylearn2/models/tests/test_kmeans.py
@@ -0,0 +1,20 @@
+import numpy as np
+
+from pylearn2.datasets.dense_design_matrix import DenseDesignMatrix
+from pylearn2.models.kmeans import KMeans
+from pylearn2.train import Train
+
+
+def test_kmeans():
+    X = np.random.random(size=(100, 10))
+    Y = np.random.randint(5, size=(100, 1))
+
+    dataset = DenseDesignMatrix(X, y=Y)
+
+    model = KMeans(
+        k=5,
+        nvis=10
+    )
+
+    train = Train(model=model, dataset=dataset)
+    train.main_loop()
Repository: lisa-lab/pylearn2

Issue:
MaxoutLocalC01B constructor doesn't call parent constructor
in [here](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/maxout.py#L1088).
Am I correct to assume this is a bug? Shall I make a pull request?

``` python
        assert (pool_shape is None) == (pool_stride is None)
        super(MaxoutLocalC01B, self).__init__()

        detector_channels = num_channels * num_pieces

        self.__dict__.update(locals())
        del self.self
```



Discussion:
And while I'm at it, how about changing [this line](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/linear/local_c01b.py#L61) from this

``` python
logger.warning("partial_sum argument ignored")
```

 to this:

``` python
if partial_sum is not None:
    raise NotImplementedError("partial_sum not implemented in pylearn2.linear.local_c01b.Local")
```

or to this:

``` python
if partial_sum is not None:
    logger.warning("partial_sum argument ignored")
```

Yes, this is a bug. A pull request would be welcome :)
Since Travis does not run on a machine with GPU, could you manually launch `nosetests` or `theano-nose` on the updated code to make sure nothing breaks?

Regarding the warning, since `partial_sum` has no effect on the numerical result (up to precision), I don't think an error is really needed.
However, I'm all for removing the warning when it is unnecessary.


Resolving patch:
diff --git a/pylearn2/linear/local_c01b.py b/pylearn2/linear/local_c01b.py
--- a/pylearn2/linear/local_c01b.py
+++ b/pylearn2/linear/local_c01b.py
@@ -58,7 +58,8 @@ def __init__(self, filters, image_shape, input_groups,
         """TODO: Local ignores partial_sum argument,
                  figure out how James' code controls it"""
 
-        logger.warning("partial_sum argument ignored")
+        if partial_sum is not None:
+            logger.warning("partial_sum argument ignored")
 
         LocalDot.__init__(self,
             filters=filters,
diff --git a/pylearn2/models/maxout.py b/pylearn2/models/maxout.py
--- a/pylearn2/models/maxout.py
+++ b/pylearn2/models/maxout.py
@@ -1086,6 +1086,7 @@ def __init__(self,
                  kernel_stride=(1, 1)):
 
         assert (pool_shape is None) == (pool_stride is None)
+        super(MaxoutLocalC01B, self).__init__()
 
         detector_channels = num_channels * num_pieces
 


Test patch:
Repository: lisa-lab/pylearn2

Issue:
Improving mapping.nest error message
If you pass mapping.nest something that is a list rather than a tuple, you get a completely unrelated error message:

```
  File "/usr/local/google/home/goodfellow/pylearn2/pylearn2/utils/data_specs.py", line 273, in nest
    assert self.n_unique_specs == 1
AssertionError
```



Discussion:
I don't really understand this portion of the codebase so @lamblin should probably comment on this.


Resolving patch:
diff --git a/pylearn2/utils/data_specs.py b/pylearn2/utils/data_specs.py
--- a/pylearn2/utils/data_specs.py
+++ b/pylearn2/utils/data_specs.py
@@ -3,6 +3,7 @@
 
 See :ref:`data_specs` for a high level overview of the relevant concepts.
 """
+from collections import Sized
 from pylearn2.space import CompositeSpace, NullSpace, Space
 from pylearn2.utils import safe_zip
 
@@ -64,7 +65,7 @@ def _fill_mapping(self, space, source):
 
         elif not isinstance(space, CompositeSpace):
             # Space is a simple Space, source should be a simple source
-            if isinstance(source, tuple):
+            if isinstance(source, (tuple, list)):
                 source, = source
 
             # If (space, source) has not already been seen, insert it.
@@ -118,7 +119,7 @@ def _fill_flat(self, nested, mapping, rval):
         if isinstance(mapping, int):
             # "nested" should actually be a single element
             idx = mapping
-            if isinstance(nested, tuple):
+            if isinstance(nested, (tuple, list)):
                 if len(nested) != 1:
                     raise ValueError("When mapping is an int, we expect "
                             "nested to be a single element. But mapping is "
@@ -179,7 +180,7 @@ def flatten(self, nested, return_tuple=False):
         # else, return something close to the type of nested
         if len(rval) == 1:
             return rval[0]
-        if isinstance(nested, tuple):
+        if isinstance(nested, (tuple, list)):
             return tuple(rval)
         elif isinstance(nested, Space):
             return CompositeSpace(rval)
@@ -205,7 +206,7 @@ def _make_nested_tuple(self, flat, mapping):
         if isinstance(mapping, int):
             # We are at a leaf of the tree
             idx = mapping
-            if isinstance(flat, tuple):
+            if isinstance(flat, (tuple, list)):
                 assert 0 <= idx < len(flat)
                 return flat[idx]
             else:
@@ -265,7 +266,7 @@ def nest(self, flat):
                 assert self.n_unique_specs == 1
             return self._make_nested_space(flat, self.spec_mapping)
         else:
-            if isinstance(flat, tuple):
+            if isinstance(flat, (list, tuple)):
                 assert len(flat) == self.n_unique_specs
             else:
                 # flat is not iterable, this is valid only if spec_mapping


Test patch:
Repository: lisa-lab/pylearn2

Issue:
[bug] _validate_impl for Conv2DSpace
[This line](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/space/__init__.py#L1968):

``` python
        if isinstance(batch, theano.gof.Variable):
```

should be

``` python
        if not is_numeric:
```

right?



Discussion:
Yes, that would make more sense.


Resolving patch:
diff --git a/pylearn2/space/__init__.py b/pylearn2/space/__init__.py
--- a/pylearn2/space/__init__.py
+++ b/pylearn2/space/__init__.py
@@ -1965,7 +1965,7 @@ def _validate_impl(self, is_numeric, batch):
         # checks batch.type against self.dtype
         super(Conv2DSpace, self)._validate_impl(is_numeric, batch)
 
-        if isinstance(batch, theano.gof.Variable):
+        if not is_numeric:
             if isinstance(batch, theano.sparse.SparseVariable):
                 raise TypeError("Conv2DSpace cannot use SparseVariables, "
                                 "since as of this writing (28 Dec 2013), "


Test patch:
Repository: lisa-lab/pylearn2

Issue:
[bug] print_monitor_cv.py model not iterable
I've tried `print_monitor_cv.py model.pkl` but I've got

```
Traceback (most recent call last):
  File "~/pylearn2/pylearn2/scripts/print_monitor_cv.py", line 84, in <module>
    main(**vars(args))
  File "~/pylearn2/pylearn2/scripts/print_monitor_cv.py", line 38, in main
    for model in list(this_models):
TypeError: 'MLP' object is not iterable
```

so I changed [this part](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/scripts/print_monitor_cv.py#L38):

``` python
        this_models = serial.load(filename)
        for model in list(this_models):
            # ...
```

to

``` python
        # ....
        this_models = serial.load(filename)

        try:
            this_models = list(this_models)
        except TypeError:
            this_models = [this_models]

        for model in this_models:
            # ...
```

PR?



Discussion:
At first, I was not sure why you would use `print_monitor_cv.py` on a pkl file with only one model inside, but it makes sense if you specify more than one pkl, and there is already a special case when there is only one model at all.
Then, I think such a PR would be a good idea. Instead of relying on `TypeError` to check if `this_models` is iterable or not, maybe you can use something like:

``` python
if not isinstance(this_models, collections.Iterable):
    this_models = [this_models]
for model in this_models:
    ...
```

Ask forgiveness not permission? :)
Ok, I'll do that.


Resolving patch:
diff --git a/pylearn2/scripts/print_monitor_cv.py b/pylearn2/scripts/print_monitor_cv.py
--- a/pylearn2/scripts/print_monitor_cv.py
+++ b/pylearn2/scripts/print_monitor_cv.py
@@ -13,6 +13,7 @@
 __maintainer__ = "Steven Kearnes"
 
 import argparse
+from collections import Iterable
 import numpy as np
 
 from pylearn2.utils import serial
@@ -35,6 +36,8 @@ def main(models, all=False):
     values = {}
     for filename in np.atleast_1d(models):
         this_models = serial.load(filename)
+        if not isinstance(this_models, Iterable):
+            this_models = [this_models]
         for model in list(this_models):
             monitor = model.monitor
             channels = monitor.channels


Test patch:
Repository: TileDB-Inc/TileDB-Py

Issue:
Skip cmake requirement if the cmake command is already available
Should be a simple check to see if `cmake` executable is available before adding it.
setup.py --tiledb arg not handled correctly
E.g. `setup.py build_ext --tiledb=/Users/tyler/tiledb/TileDB/dist/lib` gives this error:

```
RuntimeError: Could not find given --tiledb library path(s): /Users/tyler/tiledb/TileDB/dist/lib/lib/libtiledb.dylib
```

But then `setup.py build_ext --tiledb=/Users/tyler/tiledb/TileDB/dist` gives:

```
RuntimeError: Could not find given --tiledb library path(s): /Users/tyler/tiledb/TileDB/dist/libtiledb.dylib
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -268,15 +268,30 @@ def run(self):
         return bdist_egg_cmd
 
 
+def cmake_available():
+    """
+    Checks whether CMake command is available and >= version 3.3.
+    :return:
+    """
+    try:
+        output = subprocess.check_output(['cmake', '--version']).split()
+        version = output[2].decode('utf-8').split('.')
+        return int(version[0]) >= 3 and int(version[1]) >= 3
+    except:
+        return False
+
+
 def setup_requires():
     req = ['cython>=0.27',
            'numpy>=1.7',
            'setuptools>=18.0',
            'setuptools_scm>=1.5.4',
            'wheel>=0.30']
-    tiledb_path = [TILEDB_PATH] if TILEDB_PATH != '' else []
-    if not libtiledb_exists(tiledb_path):
+    # Add cmake requirement if libtiledb is not found and cmake is not available.
+    tiledb_lib_dir = [os.path.join(TILEDB_PATH, 'lib')] if TILEDB_PATH != '' else []
+    if not libtiledb_exists(tiledb_lib_dir) and not cmake_available():
         req.append('cmake>=3.11.0')
+
     return req
 
 


Test patch:
Repository: TileDB-Inc/TileDB-Py

Issue:
ignore deprecated C API warnings when cythonizing / building extension lib 
We need to support deprecated C api's to preserve backwards compatibility, these should be turned off by passing the appropriate compilation flag.


Discussion:
Are you getting an error, or do you mean to suppress the warnings? AFAICT the defaults don't enable `Werror` on mac (builds fine for me, with deprecation warnings), but might be different on linux.
It builds, but the warnings here are not very relevant as we know that these api's are deprecated.  This is not pressing IMO, just it might be a little disconcerting to have a wall of warnings show up when building from source.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -410,8 +410,13 @@ def setup_requires():
 if ver < (3,):
     TESTS_REQUIRE.extend(["unittest2", "mock"])
 
-# Globals variables
-CXXFLAGS = os.environ.get("CXXFLAGS", "-std=c++11" if not is_windows() else "").split()
+# Global variables
+CXXFLAGS = os.environ.get("CXXFLAGS", "").split()
+if not is_windows():
+  CXXFLAGS.append("-std=c++11")
+  if not TILEDB_DEBUG_BUILD:
+    CXXFLAGS.append("-Wno-deprecated-declarations")
+
 LFLAGS = os.environ.get("LFLAGS", "").split()
 
 # Allow setting (lib) TileDB directory if it is installed on the system


Test patch:
Repository: TileDB-Inc/TileDB-Py

Issue:
from_pandas throws Exception "Tile extent exceeds dimension domain range"
Connected to pydev debugger (build 201.8743.12)
Traceback (most recent call last):
  File "tiledb/libtiledb.pyx", line 2569, in tiledb.libtiledb.Dim.__init__
  File "tiledb/libtiledb.pyx", line 483, in tiledb.libtiledb.check_error
  File "tiledb/libtiledb.pyx", line 479, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 464, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: [TileDB::Dimension] Error: Tile extent check failed; Tile extent exceeds dimension domain range

The pandas dataframe is very simple, it has a MultiIndex of `date` and `instruments`. The only column is `Value`.

 ```
                                                                              Value
date              instruments                                                
2000-10-23          600795.SH                                              -1.0
                     603758.SH                                              -1.0
                     600538.SH                                              -1.0
                     002249.SZ                                              -1.0
                     300588.SZ                                              -1.0
```






Discussion:

Resolving patch:
diff --git a/tiledb/dataframe_.py b/tiledb/dataframe_.py
--- a/tiledb/dataframe_.py
+++ b/tiledb/dataframe_.py
@@ -257,7 +257,7 @@ def dim_for_column(ctx, name, dim_info, col, tile=None, full_domain=False,
         dim_max = np.max(col_values)
 
     if not dim_info.dtype in (np.bytes_, np.unicode):
-        if np.issubdtype(dtype, np.integer):
+        if np.issubdtype(dtype, np.integer) or dtype.kind == 'M':
             dim_range = np.uint64(np.abs(np.uint64(dim_max) - np.uint64(dim_min)))
             # we can't make a tile larger than the dimension range
             if dim_range < tile:


Test patch:
Repository: TileDB-Inc/TileDB-Py

Issue:
Four components should be three components?
In the recently created example "writing_dense_rgb.py" there is this fragment:
https://github.com/TileDB-Inc/TileDB-Py/blob/75ddcf56ed80ba5e1a1237b7e527ec4fbd87abb9/examples/writing_dense_rgb.py#L56-L57

It says four int32 components where it seems like it should be three int32 components. After all the values of the attribute are RGB and not RGBA.


Discussion:

Resolving patch:
diff --git a/examples/writing_dense_rgb.py b/examples/writing_dense_rgb.py
--- a/examples/writing_dense_rgb.py
+++ b/examples/writing_dense_rgb.py
@@ -53,7 +53,7 @@ def create_array():
         ),
     )
 
-    # create multi-component attribute with four int32 components
+    # create multi-component attribute with three int32 components
     attr = tiledb.Attr(dtype=np.dtype("i4, i4, i4"))
 
     schema = tiledb.ArraySchema(domain=domain, sparse=False, attrs=[attr])


Test patch:
Repository: TileDB-Inc/TileDB-Py

Issue:
Sparse writing in Dense Arrays is not working as documented.
The feature described in https://docs.tiledb.com/main/solutions/tiledb-embedded/api-usage/writing-arrays/writing-sparse-cells does not work for Dense Arrays, only for SparseArrays. It raises the following error:

```python
...
    A[m.row, m.col] = m.data
  File "tiledb/libtiledb.pyx", line 4829, in tiledb.libtiledb.DenseArrayImpl.__setitem__
  File "tiledb/libtiledb.pyx", line 3129, in tiledb.libtiledb.index_domain_subarray
IndexError: invalid index type: <class 'numpy.ndarray'>
```
 I just compiled the dev of TileDB and TileDB-Py under conda with python 3.8.5 on a Debian sid.


Discussion:

Resolving patch:
diff --git a/tiledb/dataframe_.py b/tiledb/dataframe_.py
--- a/tiledb/dataframe_.py
+++ b/tiledb/dataframe_.py
@@ -8,6 +8,7 @@
 
 import numpy as np
 import tiledb
+from tiledb import libtiledb
 from tiledb import TileDBError
 
 
@@ -645,7 +646,7 @@ def from_pandas(uri, dataframe, **kwargs):
                         coords.append(dataframe.index.get_level_values(k))
 
                 # TODO ensure correct col/dim ordering
-                A._setitem_impl(tuple(coords), write_dict, nullmaps)
+                libtiledb._setitem_impl_sparse(A, tuple(coords), write_dict, nullmaps)
 
             else:
                 if row_start_idx is None:


Test patch:
diff --git a/tiledb/tests/test_libtiledb.py b/tiledb/tests/test_libtiledb.py
--- a/tiledb/tests/test_libtiledb.py
+++ b/tiledb/tests/test_libtiledb.py
@@ -1594,6 +1594,39 @@ def test_missing_schema_error(self):
         with self.assertRaises(tiledb.TileDBError):
             tiledb.DenseArray(uri)
 
+    def test_sparse_write_to_dense(self):
+        class AssignAndCheck:
+            def __init__(self, outer, *shape):
+                self.outer = outer
+                self.shape = shape
+
+            def __setitem__(self, s, v):
+                A = np.random.rand(*self.shape)
+
+                uri = self.outer.path(
+                    f"sparse_write_to_dense{random.randint(0,np.uint64(-1))}"
+                )
+
+                tiledb.from_numpy(uri, A).close()
+                with tiledb.open(uri, "w") as B:
+                    B[s] = v
+
+                A[s] = v
+                with tiledb.open(uri) as B:
+                    assert_array_equal(A, B[:])
+
+        D = AssignAndCheck(self, 5, 5)
+        D[np.array([1, 2]), np.array([0, 0])] = np.array([0, 2])
+
+        d1 = np.array([3, 1, 2, 3, 4])
+        d2 = np.array([0, 1, 1, 2, 4])
+        data = np.array([99, 10, 20, 30, 40])
+        D[d1, d2] = data
+
+        # TODO fix broadcast assignment, expand coverage
+        # D[1, d2] = data
+        # D[d1, 1]
+
 
 class DenseVarlen(DiskTestCase):
     def test_varlen_write_bytes(self):
Repository: TileDB-Inc/TileDB-Py

Issue:
DataFrame `from_csv` doesn't support alternative s3-compatible backend
Whenever we use `.from_csv` for whatever backend (alternate s3 endpoint, minio), the custom configuration that works on usual `Array` operation doesn't work in this case.

https://github.com/TileDB-Inc/TileDB-Py/blob/24d5b3bb0869ef4d2f0d0c64c494bed037754949/tiledb/dataframe_.py#L867-L870

Here, I saw that `tiledb.array_exists(uri)` isn't under scope_ctx(ctx) which cause all `s3://` uri to call AmazonS3 instead of user-configured custom backend.

```
File ".venv/lib/python3.11/site-packages/tiledb/dataframe_.py", line 868, in from_csv
    if mode != "append" and tiledb.array_exists(uri):
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/tiledb/highlevel.py", line 149, in array_exists
    if tiledb.object_type(uri) != "array":
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "tiledb/libtiledb.pyx", line 3506, in tiledb.libtiledb.object_type
  File "tiledb/libtiledb.pyx", line 356, in tiledb.libtiledb.check_error
  File "tiledb/libtiledb.pyx", line 350, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 335, in tiledb.libtiledb._raise_tiledb_error
tiledb.cc.TileDBError: [TileDB::S3] Error: Error while listing with prefix 's3://localhost:9999/test-bucket/sparse-sample/__schema/' and delimiter '/'[Error Type: 15] [HTTP Response Code: 403] [Exception: AccessDenied] [Remote IP: 54.231.225.66] [Request ID: P7SBVXAXY300G9P9] [Headers: 'content-type' = 'application/xml' 'date' = 'Wed, 15 Nov 2023 06:11:47 GMT' 'server' = 'AmazonS3' 'transfer-encoding' = 'chunked' 'x-amz-bucket-region' = 'us-east-1' 'x-amz-id-2' = 'jSRRHRQVDXOvnZ9aMxXBeL4vq5yp9wm3VErfm8xyS9iHkozkqKeFgQzK5t1JPfxh/YSBkXfYdLo=' 'x-amz-request-id' = 'P7SBVXAXY300G9P9'] : Access Denied
```

# Environment
```
Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
>>> tiledb.version()
(0, 23, 4)
```
OS: Debian-bookworm aarch64


Discussion:

Resolving patch:
diff --git a/tiledb/dataframe_.py b/tiledb/dataframe_.py
--- a/tiledb/dataframe_.py
+++ b/tiledb/dataframe_.py
@@ -865,8 +865,9 @@ def from_csv(uri: str, csv_file: Union[str, List[str]], **kwargs):
         elif mode not in ["ingest", "append"]:
             raise tiledb.TileDBError("Invalid mode specified ('{}')".format(mode))
 
-    if mode != "append" and tiledb.array_exists(uri):
-        raise tiledb.TileDBError("Array URI '{}' already exists!".format(uri))
+    with tiledb.scope_ctx(ctx):
+        if mode != "append" and tiledb.array_exists(uri):
+            raise tiledb.TileDBError("Array URI '{}' already exists!".format(uri))
 
     # this is a pandas pass-through argument, do not pop!
     chunksize = kwargs.get("chunksize", None)


Test patch: