internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: superduper-io/superduper

Issue:
[MISC] The cosine method is slow to compute.
The Cosine method is slow in computation because it involves data transformation for the vector matrix every time, which results in significant time consumption.

```python
def cosine(x, y):
    '''
    Cosine similarity function for vector search
    '''
    x = x.astype(float)
    y = y.astype(float)
    x = x / numpy.linalg.norm(x, axis=1)[:, None]
    y = y / numpy.linalg.norm(y, axis=1)[:, None]
    return dot(x, y)

```


we need to preprocess all the incoming matrices of cosine.


Discussion:
**Suggestion**: handle conversion and normalization in preparation.
**Care required**: if we use preparation to handle `cosine`, we need to make sure that data copied across to vector-index in further updates (task: `copy_vectors`) is also normalized.


There is a problem in this approach

If we store normalised version of data in vector database
and our measure is selected as cosine 

but if user changes the measure later, then this normalised vectors will work?


But I guess When we create vector search index 
it is fixed!, hence won't be a issue though

Resolving patch:
diff --git a/superduperdb/vector_search/base.py b/superduperdb/vector_search/base.py
--- a/superduperdb/vector_search/base.py
+++ b/superduperdb/vector_search/base.py
@@ -176,10 +176,9 @@ def cosine(x, y):
     '''
     Cosine similarity function for vector search
     '''
-    x = x.astype(float)
-    y = y.astype(float)
     x = x / numpy.linalg.norm(x, axis=1)[:, None]
-    y = y / numpy.linalg.norm(y, axis=1)[:, None]
+    # y which implies all vectors in vectordatabase
+    # has normalized vectors.
     return dot(x, y)
 
 
diff --git a/superduperdb/vector_search/in_memory.py b/superduperdb/vector_search/in_memory.py
--- a/superduperdb/vector_search/in_memory.py
+++ b/superduperdb/vector_search/in_memory.py
@@ -31,6 +31,10 @@ def __init__(
         self._cache: t.Sequence[VectorItem] = []
         self._CACHE_SIZE = 10000
 
+        self.measure = measure
+        if isinstance(measure, str):
+            self.measure = measures[measure]
+
         if h is not None:
             assert index is not None
             self._setup(h, index)
@@ -39,17 +43,19 @@ def __init__(
             self.index = None
             self.lookup = None
 
-        self.measure = measure
-        if isinstance(measure, str):
-            self.measure = measures[measure]
-
         self.identifier = identifier
 
     def __len__(self):
         return self.h.shape[0]
 
     def _setup(self, h, index):
-        self.h = numpy.array(h) if not isinstance(h, numpy.ndarray) else h
+        h = numpy.array(h) if not isinstance(h, numpy.ndarray) else h
+
+        if self.measure == 'cosine':
+            # Normalization is required for cosine, hence preparing
+            # all vectors in advance.
+            h = h / numpy.linalg.norm(h, axis=1)[:, None]
+        self.h = h
         self.index = index
         self.lookup = dict(zip(index, range(len(index))))
 


Test patch:
Repository: Health-Informatics-UoN/Carrot-Mapper

Issue:
Remove Django view code that queries data
- [x]  ScanReportListView
- [x]  StructuralMappingTableListView
- [x]  (double check for any more!)

(Views that get data by the API, but are duplicating calls, as the rendered data is all driven by the API. Remove the code that does this)


Discussion:

Resolving patch:
diff --git a/app/api/mapping/views.py b/app/api/mapping/views.py
--- a/app/api/mapping/views.py
+++ b/app/api/mapping/views.py
@@ -88,44 +88,10 @@ def update_scanreport_table_page(request, sr, pk):
 @method_decorator(login_required, name="dispatch")
 class ScanReportListView(ListView):
     model = ScanReport
-    # order the scanreports now so the latest is first in the table
-    ordering = ["-created_at"]
-
-    # handle and post methods
-    # so far just handle a post when a button to click to hide/show a report
-    def post(self, request, *args, **kwargs):
-        # obtain the scanreport id from the buttont that is clicked
-        _id = request.POST.get("scanreport_id")
-        if _id is not None:
-            # obtain the scan report based on this id
-            report = ScanReport.objects.get(pk=_id)
-            # switch hidden True -> False, or False -> True, if clicked
-            report.hidden = not report.hidden
-            # update the model
-            report.save()
-        # return to the same page
-        return redirect(request.META["HTTP_REFERER"])
-
-    def get_context_data(self, **kwargs):
-        # Call the base implementation first to get a context
-        context = super().get_context_data(**kwargs)
-        # add the current user to the context
-        # this is needed so the hide/show buttons can be only turned on
-        # by whoever created the report
-        context["current_user"] = self.request.user
-        context["filterset"] = self.filterset
-        return context
 
     def get_queryset(self):
-        search_term = self.request.GET.get("filter", None)
-        qs = super().get_queryset()
-        if search_term == "archived":
-            qs = qs.filter(hidden=True)
-            self.filterset = "Archived"
-        else:
-            qs = qs.filter(hidden=False)
-            self.filterset = "Active"
-        return qs
+        # No data is passed to the view, it is all API fetched.
+        return ScanReport.objects.none()
 
 
 @method_decorator(login_required, name="dispatch")


Test patch:
Repository: Health-Informatics-UoN/Carrot-Mapper

Issue:
Datasets Index
Example: `/datasets/`

Frontend Tasks:

- [x] Recreate the existing UI
- [x] Define types in Typescript for what is needed
- [x] Add pagination
- [x] Add filtering
- [x] Add sorting
- [x] Review

Backend Tasks:

- [x] New API endpoint
- [x] With pagination, filtering, and sorting
- [x] Testing 
- [x] Add the URLS to proxy it 
- [x] Review 


Discussion:

Resolving patch:
diff --git a/app/api/api/views.py b/app/api/api/views.py
--- a/app/api/api/views.py
+++ b/app/api/api/views.py
@@ -418,12 +418,14 @@ class DatasetAndDataPartnerListView(generics.ListAPIView):
 
     serializer_class = DatasetAndDataPartnerViewSerializer
     pagination_class = CustomPagination
-    filter_backends = [DjangoFilterBackend]
+    filter_backends = [DjangoFilterBackend, OrderingFilter]
+    ordering_fields = ["id", "name", "created_at", "visibility", "data_partner"]
     filterset_fields = {
         "id": ["in"],
-        "data_partner": ["in", "exact"],
         "hidden": ["in", "exact"],
+        "name": ["in", "icontains"],
     }
+    ordering = "-created_at"
 
     def get_queryset(self):
         """
diff --git a/app/api/proxy/urls.py b/app/api/proxy/urls.py
--- a/app/api/proxy/urls.py
+++ b/app/api/proxy/urls.py
@@ -10,6 +10,11 @@
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/scanreports"),
         name="scan-report-list",
     ),
+    re_path(
+        "datasets/?(?P<path>.*)$",
+        ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/datasets"),
+        name="datasets-list",
+    ),
     re_path(
         "_next/(?P<path>.*)$",
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/_next"),


Test patch:
Repository: Health-Informatics-UoN/Carrot-Mapper

Issue:
💡 Change wording of SR list view filter box
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is your proposal related to a problem or functionality gap?

_No response_

### Describe your proposal

Please consider changing the wording in the filter box (currently "Filter by dataset") to more closely fit the true function, which is to filter on the "name" column, not the "dataset" column. I appreciate that this is somewhat confusing on the BE given the naming of fields in the ScanReport model 😆 

### Describe alternatives you've considered

_No response_

### I'm part of a Project Team

_No response_

### Anything else?

_No response_

### Are you willing to contribute to developing this feature?

None


Discussion:
Yep this is due to the backend model which has to reflected in the API. But the fix is to be able to override this in the data table component with a prop. Should be straightforward to fix.

Resolving patch:
diff --git a/app/api/api/views.py b/app/api/api/views.py
--- a/app/api/api/views.py
+++ b/app/api/api/views.py
@@ -365,7 +365,11 @@ class ScanReportListViewSetV2(ScanReportListViewSet):
     """
 
     filter_backends = [DjangoFilterBackend, OrderingFilter]
-    filterset_fields = {"hidden": ["exact"], "dataset": ["in", "icontains"]}
+    filterset_fields = {
+        "hidden": ["exact"],
+        "dataset": ["in", "icontains"],
+        "status": ["in"],
+    }
     ordering_fields = ["id", "name", "created_at", "dataset", "data_partner"]
     pagination_class = CustomPagination
     ordering = "-created_at"


Test patch:
Repository: Health-Informatics-UoN/Carrot-Mapper

Issue:
Datasets detail - list of scan reports



Discussion:

Resolving patch:
diff --git a/app/api/api/views.py b/app/api/api/views.py
--- a/app/api/api/views.py
+++ b/app/api/api/views.py
@@ -381,6 +381,7 @@ class ScanReportListViewSetV2(ScanReportListViewSet):
         "hidden": ["exact"],
         "dataset": ["in", "icontains"],
         "status": ["in"],
+        "parent_dataset": ["exact"],
     }
     ordering_fields = ["id", "name", "created_at", "dataset", "data_partner"]
     pagination_class = CustomPagination
diff --git a/app/api/proxy/urls.py b/app/api/proxy/urls.py
--- a/app/api/proxy/urls.py
+++ b/app/api/proxy/urls.py
@@ -30,6 +30,11 @@
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/datasets"),
         name="datasets-list",
     ),
+    re_path(
+        r"^datasets/(?P<path>\d+)/?$",
+        ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/datasets"),
+        name="datasets-scanreports-list",
+    ),
     re_path(
         "_next/(?P<path>.*)$",
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/_next"),


Test patch:
Repository: Health-Informatics-UoN/Carrot-Mapper

Issue:
Scan Report Edit Table



Discussion:

Resolving patch:
diff --git a/app/api/proxy/urls.py b/app/api/proxy/urls.py
--- a/app/api/proxy/urls.py
+++ b/app/api/proxy/urls.py
@@ -25,6 +25,11 @@
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/scanreports/"),
         name="scan-report-values",
     ),
+    re_path(
+        r"^scanreports/(?P<path>\d+/tables/\d+/update)/$",
+        ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/scanreports/"),
+        name="scan-report-edit-table",
+    ),
     re_path(
         r"^datasets/(?P<path>(?![\d/]).*)$",
         ProxyView.as_view(upstream=f"{settings.NEXTJS_URL}/datasets"),


Test patch:
Repository: neptune-ai/neptune-client

Issue:
create_experiment() fails on windows 10
Hi there, 

I enjoy neptune very much and on my macbook everything works fine. But when I run the same code on my Windows 10 machine, I get an error when calling create_experiment().

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\neptune\__init__.py", line 177, in create_experiment
    notebook_id=notebook_id
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\neptune\projects.py", line 400, in create_experiment
    click.echo(str(experiment.id))
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\click\utils.py", line 218, in echo
    file = _default_text_stdout()
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\click\_compat.py", line 675, in func
    rv = wrapper_func()
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\click\_compat.py", line 436, in get_text_stdout
    rv = _get_windows_console_stream(sys.stdout, encoding, errors)
  File "C:\ProgramData\Anaconda3\envs\rl_insurance\lib\site-packages\click\_winconsole.py", line 295, in _get_windows_console_stream
    func = _stream_factories.get(f.fileno())
AttributeError: 'StdOutWithUpload' object has no attribute 'fileno'`

It happens when I run:

`import neptune `
`import cfg`
`neptune.init(api_token=cfg.neptune_token, project_qualified_name=cfg.neptune_project_name) `
`neptune.create_experiment()`

I run it in conda environments both times.



Discussion:
Hi @valko073!

Thanks for letting us know. We'll take a look in the next few days.

Resolving patch:
diff --git a/neptune/internal/streams/stdstream_uploader.py b/neptune/internal/streams/stdstream_uploader.py
--- a/neptune/internal/streams/stdstream_uploader.py
+++ b/neptune/internal/streams/stdstream_uploader.py
@@ -41,6 +41,9 @@ def isatty(self):
     def flush(self):
         self._stream.flush()
 
+    def fileno(self):
+        return self._stream.fileno()
+
 
 class StdOutWithUpload(StdStreamWithUpload):
 


Test patch:
Repository: neptune-ai/neptune-client

Issue:
experiment.get_system_properties() doesn't return "hostname"
I think there is some regression. For recent experiments `experiment.get_properties()` return either an empty dictionary or `{'key1': 'value1', 'key2': '17', 'key3': 'other-value'}` (whatever that is) in case of the sandbox project.

For older experiments, I still can get the properties. 

This is probably a backend issue but there is no better place to put it.


Discussion:

Resolving patch:
diff --git a/neptune/internal/streams/channel_writer.py b/neptune/internal/streams/channel_writer.py
--- a/neptune/internal/streams/channel_writer.py
+++ b/neptune/internal/streams/channel_writer.py
@@ -16,8 +16,8 @@
 
 from __future__ import unicode_literals
 
+from datetime import datetime
 import re
-import time
 
 from neptune.internal.channels.channels import ChannelNamespace, ChannelValue, ChannelType
 
@@ -26,7 +26,7 @@ class ChannelWriter(object):
     __SPLIT_PATTERN = re.compile(r'[\n\r]{1,2}')
 
     def __init__(self, experiment, channel_name, channel_namespace=ChannelNamespace.USER):
-        self.time_started_ms = time.time() * 1000
+        self._time_started = experiment.get_system_properties()['created']
         self._experiment = experiment
         self._channel_name = channel_name
         self._channel_namespace = channel_namespace
@@ -40,7 +40,7 @@ def write(self, data):
         lines = self.__SPLIT_PATTERN.split(self._data)
         for line in lines[:-1]:
             value = ChannelValue(
-                x=time.time() * 1000 - self.time_started_ms,
+                x=(datetime.now(tz=self._time_started.tzinfo) - self._time_started).total_seconds() * 1000,
                 y=dict(text_value=str(line)),
                 ts=None
             )


Test patch:
diff --git a/tests/neptune/test_channel_writer.py b/tests/neptune/test_channel_writer.py
--- a/tests/neptune/test_channel_writer.py
+++ b/tests/neptune/test_channel_writer.py
@@ -15,6 +15,7 @@
 #
 
 import unittest
+from datetime import datetime
 
 from mock import MagicMock
 
@@ -26,6 +27,7 @@ class TestChannelWriter(unittest.TestCase):
     def test_write_data_to_channel_writer(self):
         # given
         experiment = MagicMock()
+        experiment.get_system_properties.return_value = {"created": datetime.now()}
         channel_name = 'a channel name'
         writer = ChannelWriter(experiment, channel_name)
Repository: neptune-ai/neptune-client

Issue:
BUG: Inconsistency in `fetch_run_table` tag selector
Hi,

according to `fetch_run_table`'s documentation:

> Only experiments that have all specified tags will match this criterion.

Thus, the query aggregator for tags should be `AND`.

https://github.com/neptune-ai/neptune-client/blob/4b164bc470278edd160b7b9b5059a32c4822b376/neptune/new/metadata_containers/project.py#L235


Discussion:

Resolving patch:
diff --git a/neptune/new/metadata_containers/project.py b/neptune/new/metadata_containers/project.py
--- a/neptune/new/metadata_containers/project.py
+++ b/neptune/new/metadata_containers/project.py
@@ -232,7 +232,7 @@ def fetch_runs_table(
                         )
                         for tag in tags
                     ],
-                    aggregator=NQLAggregator.OR,
+                    aggregator=NQLAggregator.AND,
                 )
             )
 


Test patch:
diff --git a/e2e_tests/standard/test_base.py b/e2e_tests/standard/test_base.py
--- a/e2e_tests/standard/test_base.py
+++ b/e2e_tests/standard/test_base.py
@@ -177,30 +177,30 @@ def test_add_and_remove_tags(self, container: MetadataContainer):
 
 class TestFetchTable(BaseE2ETest):
     def test_fetch_runs_table(self, environment):
-        tag = str(uuid.uuid4())
+        tag1, tag2 = str(uuid.uuid4()), str(uuid.uuid4())
 
         with neptune.init_run(project=environment.project) as run:
-            run["sys/tags"].add(tag)
+            run["sys/tags"].add(tag1)
+            run["sys/tags"].add(tag2)
             run["value"] = 12
             run.sync()
 
         with neptune.init_run(project=environment.project) as run:
-            run["sys/tags"].add(tag)
+            run["sys/tags"].add(tag2)
             run["another/value"] = "testing"
             run.sync()
 
-        # wait for the elasticsearch cache to fill
+        # wait for the cache to fill
         time.sleep(5)
 
         project = neptune.get_project(name=environment.project)
 
         runs_table = sorted(
-            project.fetch_runs_table(tag=tag).to_rows(),
+            project.fetch_runs_table(tag=[tag1, tag2]).to_rows(),
             key=lambda r: r.get_attribute_value("sys/id"),
         )
-        assert len(runs_table) == 2
+        assert len(runs_table) == 1
         assert runs_table[0].get_attribute_value("value") == 12
-        assert runs_table[1].get_attribute_value("another/value") == "testing"
 
     @pytest.mark.parametrize("container", ["model"], indirect=True)
     def test_fetch_model_versions_table(self, container: Model, environment):
Repository: google/timesketch

Issue:
Unable to get event and add event annotation when using PGSQL
When using postgresql some queries fails with DataError.

This is due to the way SQLalchemy constructs the query, and the way we pass in the searchindex name to the filter.



Discussion:

Resolving patch:
diff --git a/timesketch/api/v1/resources.py b/timesketch/api/v1/resources.py
--- a/timesketch/api/v1/resources.py
+++ b/timesketch/api/v1/resources.py
@@ -374,7 +374,8 @@ def get(self, sketch_id):
         args = self.parser.parse_args()
         sketch = Sketch.query.get_with_acl(sketch_id)
         searchindex_id = args.get(u'searchindex_id')
-        searchindex = SearchIndex.query.get(searchindex_id)
+        searchindex = SearchIndex.query.filter_by(
+            index_name=searchindex_id).first()
         event_id = args.get(u'event_id')
         indices = [t.searchindex.index_name for t in sketch.timelines]
 
@@ -428,7 +429,8 @@ def post(self, sketch_id):
             indices = [t.searchindex.index_name for t in sketch.timelines]
             annotation_type = form.annotation_type.data
             searchindex_id = form.searchindex_id.data
-            searchindex = SearchIndex.query.get(searchindex_id)
+            searchindex = SearchIndex.query.filter_by(
+                index_name=searchindex_id).first()
             event_id = form.event_id.data
             event_type = form.event_type.data
 
diff --git a/wsgi.py b/wsgi.py
--- a/wsgi.py
+++ b/wsgi.py
@@ -33,5 +33,11 @@
 #execfile(activate_virtualenv, dict(__file__=activate_virtualenv))
 
 from timesketch import create_app
+from timesketch.models import db_session
 
 application = create_app()
+
+# Remove the session after every request or app shutdown.
+@application.teardown_appcontext
+def shutdown_session(exception=None):
+    db_session.remove()


Test patch:
diff --git a/timesketch/api/v1/resources_test.py b/timesketch/api/v1/resources_test.py
--- a/timesketch/api/v1/resources_test.py
+++ b/timesketch/api/v1/resources_test.py
@@ -149,9 +149,6 @@ class EventResourceTest(BaseTest):
     """Test EventResource."""
     resource_url = u'/api/v1/sketches/1/event/'
     expected_response = {
-        u'meta': {
-            u'comments': []
-        },
         u'objects': {
             u'timestamp_desc': u'',
             u'timestamp': 1410895419859714,
@@ -172,7 +169,7 @@ def test_get_event(self):
         self.login()
         response = self.client.get(
             self.resource_url + u'?searchindex_id=test&event_id=test')
-        self.assertDictEqual(response.json, self.expected_response)
+        self.assertDictContainsSubset(self.expected_response, response.json)
         self.assert200(response)
 
     @mock.patch(
Repository: google/timesketch

Issue:
Importing of JSON timelines creates duplicate timelines with same name.
Steps to reproduce
1) command line:
echo '[
  {
    "datetime": "2012-04-12T17:24:38-08:00",
    "timestamp_desc": "Test",
    "timestamp": 1334251478000000,
    "message": "Test message"
  }
]' > test_dupe.json  
tsctl json2ts --name test_dupe --file test_dupe.json
tsctl json2ts --name test_dupe --file test_dupe.json

2) Create new sketch
3) Notice duplicate "test_dupe" timelines on list to select from.
4) Add both
5) Explore, using "*" as filter.
6) notice duplicate results.



Discussion:

Resolving patch:
diff --git a/wsgi.py b/wsgi.py
--- a/wsgi.py
+++ b/wsgi.py
@@ -37,7 +37,8 @@
 
 application = create_app()
 
-# Remove the session after every request or app shutdown.
+# pylint: disable=unused-argument
 @application.teardown_appcontext
 def shutdown_session(exception=None):
+    """Remove the database session after every request or app shutdown."""
     db_session.remove()


Test patch:
Repository: google/timesketch

Issue:
View is not saved when URL parameters are present
The user view is not saved to the database if the view is created from URL parameters.



Discussion:

Resolving patch:
diff --git a/timesketch/ui/views/sketch.py b/timesketch/ui/views/sketch.py
--- a/timesketch/ui/views/sketch.py
+++ b/timesketch/ui/views/sketch.py
@@ -171,6 +171,8 @@ def explore(sketch_id, view_id=None):
     Returns:
         Template with context.
     """
+    DEFAULT_LIMIT = 100  # Number of events to return.
+    save_view = False  # If the view should be saved to the database.
     sketch = Sketch.query.get_with_acl(sketch_id)
     sketch_timelines = [t.searchindex.index_name for t in sketch.timelines]
     view_form = SaveViewForm()
@@ -196,12 +198,11 @@ def explore(sketch_id, view_id=None):
         view = sketch.get_user_view(current_user)
 
     if not view:
-        query_filter = dict(indices=sketch_timelines)
+        query_filter = dict(indices=sketch_timelines, limit=DEFAULT_LIMIT)
         view = View(
-            user=current_user, name=u'', sketch=sketch, query_string=u'',
+            user=current_user, name=u'', sketch=sketch, query_string=u'*',
             query_filter=json.dumps(query_filter, ensure_ascii=False))
-        db_session.add(view)
-        db_session.commit()
+        save_view = True
 
     if url_query:
         view.query_string = url_query
@@ -213,6 +214,17 @@ def explore(sketch_id, view_id=None):
         if url_limit:
             query_filter[u'limit'] = url_limit
         view.query_filter = json.dumps(query_filter, ensure_ascii=False)
+        save_view = True
+
+    query_filter = json.loads(view.query_filter)
+    if not query_filter.get(u'limit', None):
+        query_filter[u'limit'] = DEFAULT_LIMIT
+        view.query_filter = json.dumps(query_filter, ensure_ascii=False)
+        save_view = True
+
+    if save_view:
+        db_session.add(view)
+        db_session.commit()
 
     return render_template(
         u'sketch/explore.html', sketch=sketch, view=view, named_view=view_id,


Test patch: