internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: voxel51/fiftyone

Issue:
[FR] Include `score` as `confidence` in BDD imports and exports
`score` is a part of the [format standard](https://github.com/ucbdrive/bdd100k/blob/master/doc/format.md) of BDD labels, and translates to `confidence` in FiftyOne. Users would benefit if this value is automatically parsed as `confidence` and exported as `score` when working with BDD datasets.


Discussion:

Resolving patch:
diff --git a/fiftyone/utils/bdd.py b/fiftyone/utils/bdd.py
--- a/fiftyone/utils/bdd.py
+++ b/fiftyone/utils/bdd.py
@@ -523,6 +523,7 @@ def _parse_frame_labels(attrs_dict):
 
 def _parse_bdd_detection(d, frame_size, extra_attrs):
     label = d["category"]
+    confidence = d.get("score", None)
 
     width, height = frame_size
     box2d = d["box2d"]
@@ -536,11 +537,17 @@ def _parse_bdd_detection(d, frame_size, extra_attrs):
     attributes = d.get("attributes", {})
     attributes = _filter_attributes(attributes, extra_attrs)
 
-    return fol.Detection(label=label, bounding_box=bounding_box, **attributes)
+    return fol.Detection(
+        label=label,
+        bounding_box=bounding_box,
+        confidence=confidence,
+        **attributes,
+    )
 
 
 def _parse_bdd_polylines(d, frame_size, extra_attrs):
     label = d["category"]
+    confidence = d.get("score", None)
 
     attributes = d.get("attributes", {})
 
@@ -563,6 +570,7 @@ def _parse_bdd_polylines(d, frame_size, extra_attrs):
             fol.Polyline(
                 label=label,
                 points=[points],
+                confidence=confidence,
                 closed=closed,
                 filled=filled,
                 **_attributes,
@@ -651,7 +659,7 @@ def _detection_to_bdd(detection, frame_size, extra_attrs):
 
     attributes = _get_attributes(detection, extra_attrs)
 
-    return {
+    d = {
         "id": None,
         "category": detection.label,
         "manualAttributes": True,
@@ -660,6 +668,11 @@ def _detection_to_bdd(detection, frame_size, extra_attrs):
         "box2d": box2d,
     }
 
+    if detection.confidence is not None:
+        d["score"] = detection.confidence
+
+    return d
+
 
 def _polyline_to_bdd(polyline, frame_size, extra_attrs):
     width, height = frame_size
@@ -677,7 +690,7 @@ def _polyline_to_bdd(polyline, frame_size, extra_attrs):
             {"types": types, "closed": polyline.closed, "vertices": vertices,}
         )
 
-    return {
+    d = {
         "id": None,
         "category": polyline.label,
         "manualAttributes": True,
@@ -686,6 +699,11 @@ def _polyline_to_bdd(polyline, frame_size, extra_attrs):
         "poly2d": poly2d,
     }
 
+    if polyline.confidence is not None:
+        d["score"] = polyline.confidence
+
+    return d
+
 
 def _get_attributes(label, extra_attrs):
     if extra_attrs == True:


Test patch:
Repository: voxel51/fiftyone

Issue:
[FR] update opencv-python-headless
### Proposal Summary
Currently this repo requires opencv-python-headless<=4.4.0.46. To cut a long story short there are no wheels available for python3.9 and I am unable to install fiftyone (I am using docker `image: jupyter/scipy-notebook:latest`). However version `4.5.3.56` is available for install without issue, and I propose updating the requirement for this dependency.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -78,6 +78,7 @@ def get_version():
         "mongoengine==0.20.0",
         "motor>=2.3,<3",
         "numpy",
+        "opencv-python-headless",
         "packaging",
         "pandas",
         "Pillow>=6.2",
@@ -96,12 +97,6 @@ def get_version():
         "universal-analytics-python3>=1.0.1,<2",
         # internal packages
         "voxel51-eta>=0.5.2,<0.6",
-        # ETA dependency - restricted to a maximum version known to provide
-        # wheels here because it tends to publish sdists several hours before
-        # wheels. When users install FiftyOne in this window, they will need to
-        # compile OpenCV from source, leading to either errors or a
-        # time-consuming installation.
-        "opencv-python-headless<=4.4.0.46",
     ],
     classifiers=[
         "Development Status :: 4 - Beta",


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] App grid view does not contain all samples
On `develop`, when I load the quickstart dataset in the App and start scrolling, the grid view exhausts before it gets to the 200th sample:

```py
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)

# Scroll to the end of the grid

# Now show the actual last sample, which differs from above
session.view = dataset.skip(len(dataset) - 1)
```



Discussion:

Resolving patch:
diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -254,9 +254,7 @@ async def get(self):
 
         samples = await foo.aggregate(
             StateHandler.sample_collection(),
-            view.skip((page - 1) * page_length)._pipeline(
-                attach_frames=True, detach_frames=False
-            ),
+            view._pipeline(attach_frames=True, detach_frames=False),
         ).to_list(page_length + 1)
         convert(samples)
 


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Force reencode flag not working
```
from fiftyone.utils import video as fov
<creating the dataset>
fov.reencode_videos(dataset, force_reencode=False)
``` 
does not ignore existing mp4 videos per the documentation, producing unwanted copies of videos.

### System information

-   **Ubuntu 18.04**
-   **source**
-   **0.14**
-   **3.8.11**
-  **ffmpeg version 4.2.4-1ubuntu0.1**


Discussion:

Resolving patch:
diff --git a/fiftyone/utils/video.py b/fiftyone/utils/video.py
--- a/fiftyone/utils/video.py
+++ b/fiftyone/utils/video.py
@@ -561,9 +561,12 @@ def _transform_videos(
                 # that all frames exist
                 if not force_reencode and os.path.exists(outpath % 1):
                     continue
-
             elif reencode:
-                outpath = os.path.splitext(inpath)[0] + ".mp4"
+                root, ext = os.path.splitext(inpath)
+                if ext.lower() != ".mp4":
+                    outpath = root + ".mp4"
+                else:
+                    outpath = inpath
             else:
                 outpath = inpath
 
@@ -660,11 +663,10 @@ def _transform_video(
             kwargs["out_opts"] = []
 
     should_reencode = (
-        fps is not None
+        force_reencode
+        or fps is not None
         or size is not None
-        or reencode
-        or in_ext != out_ext
-        or force_reencode
+        or in_ext.lower() != out_ext.lower()
     )
 
     if (inpath == outpath) and should_reencode:
@@ -681,7 +683,6 @@ def _transform_video(
     elif should_reencode:
         with etav.FFmpeg(fps=fps, size=size, **kwargs) as ffmpeg:
             ffmpeg.run(inpath, outpath, verbose=verbose)
-
     elif diff_path:
         etau.copy_file(inpath, outpath)
 


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] App label filter not working in Colab
I ran through the [quickstart](https://colab.research.google.com/github/voxel51/fiftyone-examples/blob/master/examples/quickstart.ipynb) in Colab (using `fiftyone==0.14.0`) and found an App error when trying to use a `label` filter in the App.

The error also occurred when using the same label filter from the expanded modal. However, other filters such as confidence and the alternate low-choices mode of the label filter were working as expected, so I believe this is strictly related to the autocomplete-style filter UI.

I cannot reproduce this error using `fiftyone==0.14.0` outside of colab.

<img width="1156" alt="Screen Shot 2021-11-03 at 10 54 54 AM" src="https://user-images.githubusercontent.com/25985824/140086403-9f2ab519-c595-47a9-a3b3-a7b772ace06e.png">



Discussion:

Resolving patch:
diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -377,6 +377,7 @@ async def post(self):
                     message = {"state": StateHandler.state}
 
             if event in {
+                "count_values",
                 "distinct",
                 "distributions",
                 "get_video_data",


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Image paths not ending with an extension make app fail to open
### System information

-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux
-   **FiftyOne installed from (pip or source)**: pip
-   **FiftyOne version (run `fiftyone --version`)**: 0.14.0
-   **Python version**: 3.8

### Describe the problem

If images in a dataset contain path which do not end with an extension e.g. `/path/to/image` instead of `/path/to/image.jpg`, when launching the fiftyone app, it does not load and shows a 500 error without any clear error message.

I can provide a piece of code to reproduce if really needed but I think my description is fairly straightforward.

### What areas of FiftyOne does this bug affect?

-   [x] `App`: FiftyOne application issue
-   [ ] `Core`: Core `fiftyone` Python library issue
-   [ ] `Server`: Fiftyone server issue

### Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another
member of your organization be willing to contribute a fix for this bug to the
FiftyOne codebase?

-   [ ] Yes. I can contribute a fix for this bug independently.
-   [ ] Yes. I would be willing to contribute a fix for this bug with guidance
        from the FiftyOne community.
-   [x] No. I cannot contribute a bug fix at this time.


Discussion:

Resolving patch:
diff --git a/fiftyone/server/utils.py b/fiftyone/server/utils.py
--- a/fiftyone/server/utils.py
+++ b/fiftyone/server/utils.py
@@ -127,7 +127,7 @@ def read_metadata(filepath, metadata=None):
         dict
     """
     mimetype, _ = mimetypes.guess_type(filepath)
-    if mimetype.startswith("video/"):
+    if mimetype and mimetype.startswith("video/"):
         if metadata:
             width = metadata.get("frame_width", None)
             height = metadata.get("frame_height", None)


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Unable to clone view whose parent dataset has a brain run
On `fiftyone==0.14.0`, Trying to clone a view into a dataset with a brain run:

```py
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(dataset, brain_key="img_sim")

dataset2 = dataset.limit(1).clone()
```

raises the following error:

```
ValueError: Number of sample IDs (1) does not match number of embeddings (200). You may have missing data/labels that you need to omit from your view
```



Discussion:

Resolving patch:
diff --git a/fiftyone/core/dataset.py b/fiftyone/core/dataset.py
--- a/fiftyone/core/dataset.py
+++ b/fiftyone/core/dataset.py
@@ -4715,8 +4715,8 @@ def _clone_dataset_or_view(dataset_or_view, name):
 
     clone_dataset = load_dataset(name)
 
-    # Clone run results
-    if (
+    # Clone run results (full datasets only)
+    if view is None and (
         dataset.has_annotation_runs
         or dataset.has_brain_runs
         or dataset.has_evaluations


Test patch:
Repository: voxel51/fiftyone

Issue:
Is it possible to pass the original filenames to CVAT?
When uploading the images from fiftyone to CVAT, the filenames are lost. When syncing with a dataset repository, however, they are crucial as there is no other way of getting the original filenames to the annotations. I think #1235 could possibly fix this, if it was implemented. Are there currently any other workarounds? The labeling order in a given task in CVAT is not important to me.


Discussion:

Resolving patch:
diff --git a/fiftyone/utils/cvat.py b/fiftyone/utils/cvat.py
--- a/fiftyone/utils/cvat.py
+++ b/fiftyone/utils/cvat.py
@@ -3370,7 +3370,7 @@ def upload_data(
             # IMPORTANT: CVAT organizes media within a task alphabetically by
             # filename, so we must give CVAT filenames whose alphabetical order
             # matches the order of `paths`
-            filename = "%06d%s" % (idx, os.path.splitext(path)[1])
+            filename = "%06d_%s" % (idx, os.path.basename(path))
             files["client_files[%d]" % idx] = (filename, open(path, "rb"))
 
         self.post(self.task_data_url(task_id), data=data, files=files)


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] `ImportError: cannot import name 'soft_unicode' from 'markupsafe'`
When `Jinja2<3` is installed a user will encounter this error.

```py
>>> import fiftyone as fo
Uncaught exception
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/fiftyone/service/main.py", line 43, in <module>
    from fiftyone.core.service import Service
  File "/home/user/.local/lib/python3.8/site-packages/fiftyone/__init__.py", line 25, in <module>
    from fiftyone.__public__ import *
  File "/home/user/.local/lib/python3.8/site-packages/fiftyone/__public__.py", line 172, in <module>
    from .core.session import (
  File "/home/user/.local/lib/python3.8/site-packages/fiftyone/core/session.py", line 16, in <module>
    from jinja2 import Template
  File "/home/user/.local/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/home/user/.local/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/home/user/.local/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/home/user/.local/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/home/user/.local/lib/python3.8/site-packages/markupsafe/__init__.py)
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@ def get_version():
     "Deprecated",
     "eventlet",
     "future",
-    "Jinja2",
+    "Jinja2>=3",
     "kaleido",
     "matplotlib",
     "mongoengine==0.20.0",


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Support Fortran ordered masks in the App
Currently fortran ordered masks are flipped.

```py
import fiftyone as fo
import fiftyone.zoo as foz
import numpy as np

dataset = foz.load_zoo_dataset("quickstart", max_samples=1).select_fields().clone()
sample = dataset.first()

contiguous = np.asarray([[True, False], [True, False]])
sample["contiguous"] = fo.Segmentation(mask=contiguous)
sample["fortran"] = fo.Segmentation(mask=np.asfortranarray(contiguous))
sample.save()

session = fo.Session(dataset)
```
<img width="1792" alt="flipped" src="https://user-images.githubusercontent.com/19821840/159953546-5eef71bc-d111-4667-a271-6c4e34e1b7da.png">




Discussion:

Resolving patch:
diff --git a/fiftyone/server/json_util.py b/fiftyone/server/json_util.py
--- a/fiftyone/server/json_util.py
+++ b/fiftyone/server/json_util.py
@@ -10,6 +10,7 @@
 from datetime import date, datetime
 from json import JSONEncoder
 import math
+import numpy as np
 
 from fiftyone.core.sample import Sample, SampleView
 from fiftyone.core.stages import ViewStage
@@ -33,9 +34,12 @@ def _handle_numpy_array(raw, _cls=None):
     if _cls not in _MASK_CLASSES:
         return str(fou.deserialize_numpy_array(raw).shape)
 
-    return fou.serialize_numpy_array(
-        fou.deserialize_numpy_array(raw), ascii=True
-    )
+    array = fou.deserialize_numpy_array(raw)
+
+    if np.isfortran(array):
+        array = np.ascontiguousarray(array)
+
+    return fou.serialize_numpy_array(array, ascii=True)
 
 
 def _handle_date(dt):


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Classifications label bubbles not displayed correctly
When viewing a `Classifications` field in the App, the bubble that is shown displays the `label` attribute of the `Classifications` rather than each `label` of each individual `Classification` in the `Classifications` list.

```python
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart").select_fields().clone()

sample = dataset.first()
sample["classifications"] = fo.Classifications(
    label="top_label",
    classifications=[
        fo.Classification(label="test1"),
        fo.Classification(label="test2"),
    ]
)
sample.save()

session = fo.launch_app(dataset)
```
I would expect to see `test1` and `test2` bubbles rather than `top_label`.

![image](https://user-images.githubusercontent.com/21222883/171876796-1f29d939-3d93-4aa3-8f17-eb93fd001941.png)




Discussion:

Resolving patch:
diff --git a/fiftyone/core/session/session.py b/fiftyone/core/session/session.py
--- a/fiftyone/core/session/session.py
+++ b/fiftyone/core/session/session.py
@@ -958,7 +958,9 @@ def close(self) -> None:
         if self.remote:
             return
 
-        self._client.send_event(CloseSession())
+        if self._client._connected:
+            self._client.send_event(CloseSession())
+
         self.plots.disconnect()
 
     def freeze(self) -> None:
diff --git a/fiftyone/server/query.py b/fiftyone/server/query.py
--- a/fiftyone/server/query.py
+++ b/fiftyone/server/query.py
@@ -214,6 +214,7 @@ class AppConfig:
     show_confidence: bool
     show_index: bool
     show_label: bool
+    show_skeletons: bool
     show_tooltip: bool
     timezone: t.Optional[str]
     use_frame_number: bool


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] tfrecord shard enumeration starting from 1 instead of 0
### System information

-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Arch Linux
-   **FiftyOne installed from (pip or source)**: pip
-   **FiftyOne version (run `fiftyone --version`)**: v0.16.2
-   **Python version**: 3.10.4

### Commands to reproduce

export a view or dataset using the `export(...)` method with `num_shards >1`

### Describe the problem

Currently tfrecord shards are exported with filenames of the format `filename.tfrecord-xxxxx-of-yyyyy` with `x` ranging from `1` to `y`. This works in general if imported using the `tf.data` api but it seems that the range should be `0` to `y-1`.
We are trying to convert our exported tfrecord shards to a `tfds`. This fails as [`tfds.folder_dataset.compute_split_info`](https://www.tensorflow.org/datasets/api_docs/python/tfds/folder_dataset/compute_split_info)
expects enumeration starting from `0`.
We also looked through a couple of tensorflow datasets and enumeration from one looks like the common case.

### Code to reproduce issue

### Other info / logs

### What areas of FiftyOne does this bug affect?

-   [ ] `App`: FiftyOne application issue
-   [x] `Core`: Core `fiftyone` Python library issue
-   [ ] `Server`: Fiftyone server issue

### Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another
member of your organization be willing to contribute a fix for this bug to the
FiftyOne codebase?

-   [x] Yes. I can contribute a fix for this bug independently.
-   [ ] Yes. I would be willing to contribute a fix for this bug with guidance
        from the FiftyOne community.
-   [ ] No. I cannot contribute a bug fix at this time.



Discussion:
@laserK3000 feel free to make a PR to resolve this 🤗 

Resolving patch:
diff --git a/fiftyone/utils/tf.py b/fiftyone/utils/tf.py
--- a/fiftyone/utils/tf.py
+++ b/fiftyone/utils/tf.py
@@ -249,7 +249,7 @@ def __enter__(self):
             tf_records_patt = self.tf_records_path + "-%05d-of-%05d"
             tf_records_paths = [
                 tf_records_patt % (i, self.num_shards)
-                for i in range(1, self.num_shards + 1)
+                for i in range(0, self.num_shards)
             ]
         else:
             self._num_shards = 1


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Changing session.view resets field visibility choices
On `fiftyone==0.16.2`, updating `session.view` resets any field visibility toggles I may have set (eg, unselected all label fields), and forces the defaults (all label fields visible). I don't think this used to be the case though?

This came up when I was trying to work with an interactive plot. I just wanted to see images with no labels, but every time I made a selection in the linked plot, the labels kept re-appearing, which was annoying. I don't recall facing this issue before.

Of course persisting sidebar settings is a bit tricky because views can change the label schema.


Discussion:

Resolving patch:
diff --git a/fiftyone/server/query.py b/fiftyone/server/query.py
--- a/fiftyone/server/query.py
+++ b/fiftyone/server/query.py
@@ -175,9 +175,7 @@ async def resolver(
         view = fov.DatasetView._build(ds, view or [])
         if view._dataset != ds:
             d = view._dataset._serialize()
-            dataset.id = (
-                ObjectId()
-            )  # if it is not the root dataset, change the id (relay requires it)
+            dataset.id = view._dataset._doc.id
             dataset.media_type = d["media_type"]
             dataset.sample_fields = [
                 from_dict(SampleField, s)
@@ -285,7 +283,7 @@ def serialize_dataset(dataset: fod.Dataset, view: fov.DatasetView) -> t.Dict:
     if view is not None and view._dataset != dataset:
         d = view._dataset._serialize()
         data.media_type = d["media_type"]
-        data.id = ObjectId()
+        data.id = view._dataset._doc.id
         data.sample_fields = [
             from_dict(SampleField, s)
             for s in _flatten_fields([], d["sample_fields"])


Test patch:
Repository: voxel51/fiftyone

Issue:
[BUG] Unselecting a field in the sidebar doesn't prevent tagging
In the App, when you unselect a label field and tag selected samples, then it tags all labels in all label fields, even ones that aren't selected.

For example, if you unselect `predictions` in the quickstart dataset, the tag the ground truth of the first sample, the `Labels` text changes from `+ tag 17 selected labels` to `+ tag 3 selected labels` as you untag the `predictions` field. 
![image](https://user-images.githubusercontent.com/21222883/192051958-f6ea292c-114b-49a9-aa38-8949e8fe72b7.png)

But actually applying this tag doesn't just apply it to the 3 selected labels in the ground truth, it applies it to all 17 in ground truth and predictions for that sample.
![image](https://user-images.githubusercontent.com/21222883/192052020-c6b961db-0983-402f-955f-5e605f75a195.png)



Discussion:

Resolving patch:
diff --git a/fiftyone/core/session/session.py b/fiftyone/core/session/session.py
--- a/fiftyone/core/session/session.py
+++ b/fiftyone/core/session/session.py
@@ -593,7 +593,6 @@ def plots(self, plots: t.Optional[fop.PlotManager]) -> None:
 
         self._plots = plots
 
-    @update_state()
     def refresh(self) -> None:
         """Refreshes the current App window."""
         self._client.send_event(StateUpdate(state=self._state, refresh=True))
diff --git a/fiftyone/core/state.py b/fiftyone/core/state.py
--- a/fiftyone/core/state.py
+++ b/fiftyone/core/state.py
@@ -113,9 +113,7 @@ def from_dict(cls, d, with_config=None):
         for field, value in d.get("config", {}).items():
             setattr(config, field, value)
 
-        timezone = d.get("config", {}).get("timezone", None)
-        if timezone:
-            fo.config.timezone = timezone
+        fo.config.timezone = d.get("config", {}).get("timezone", None)
 
         return cls(
             config=config,


Test patch: