anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: voxel51/fiftyone
Issue: [BUG] Boolean App filters raise error when used

Using the filter bar in the App to filter on a boolean field raises an error on `fiftyone==0.13.2`:

```py
import fiftyone as fo

sample = fo.Sample(filepath="/path/to/image.png", bool=True)

dataset = fo.Dataset()
dataset.add_sample(sample)

# Try selecting bool -> True in App
session = fo.launch_app(dataset)
```

```
Server Error

An exception has been raised by the server. Your session has been reverted to its previous state.

Traceback (most recent call last):
  File "main.py", line 271, in wrapper
    result = await func(self, *args, **kwargs)
  File "main.py", line 518, in on_message
    await event(self, **message)
  File "main.py", line 586, in on_filters_update
    await self.send_statistics(view, filters=filters, extended=True)
  File "main.py", line 1010, in send_statistics
    aggregations = fos.DatasetStatistics(view, filters).aggregations
  File "/Users/Brian/dev/env/tmp3/lib/python3.6/site-packages/fiftyone/core/state.py", line 167, in __init__
    self._aggregations = self._build(view, filters)
  File "/Users/Brian/dev/env/tmp3/lib/python3.6/site-packages/fiftyone/core/state.py", line 277, in _build
    else filters[field_name]["values"]
KeyError: 'values'
```


Changed files:
fiftyone/core/aggregations.py, fiftyone/core/state.py

diff --git a/fiftyone/core/aggregations.py b/fiftyone/core/aggregations.py
index b0018fc49..a83bfacd1 100644
--- a/fiftyone/core/aggregations.py
+++ b/fiftyone/core/aggregations.py
@@ -508,7 +508,6 @@ class CountValues(Aggregation):
         Returns:
             a dict mapping values to counts
         """
-
         if self._first is None:
             return {i["k"]: i["count"] for i in d["result"]}
 
diff --git a/fiftyone/core/state.py b/fiftyone/core/state.py
index 73098feef..061138268 100644
--- a/fiftyone/core/state.py
+++ b/fiftyone/core/state.py
@@ -265,10 +265,9 @@ class DatasetStatistics(object):
                 aggregations.append(foa.Count(field_name))
                 if _meets_type(field, (fof.IntField, fof.FloatField)):
                     aggregations.append(foa.Bounds(field_name))
-                elif _meets_type(
-                    field,
-                    (fof.BooleanField, fof.StringField, fof.ObjectIdField),
-                ):
+                elif _meets_type(field, fof.BooleanField):
+                    aggregations.append(foa.CountValues(field_name, _first=3))
+                elif _meets_type(field, (fof.StringField, fof.ObjectIdField)):
                     include = (
                         None
                         if filters is None
Repository: voxel51/fiftyone
Issue: [BUG] Refreshing browser tab exits session.wait()

Refreshing the App window in the browser causes `session.wait()` to exit, but the intention is that `session.wait()` should only exit when the App window is actually closed.

This is worth addressing because refreshing the App window via ctrl + R is a common practice for App users.

```py
import fiftyone as fo 

session = fo.launch_app()

# should block until App is closed, but ctrl + R also breaks
session.wait()  
```


Changed files:
fiftyone/server/main.py

diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
index c07fa75b5..97d8b71d3 100644
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -206,6 +206,7 @@ def _catch_errors(func):
 
 _notebook_clients = {}
 _deactivated_clients = set()
+_DISCONNECT_TIMEOUT = 1  # seconds
 
 
 class PollingHandler(tornado.web.RequestHandler):
@@ -421,8 +422,13 @@ class StateHandler(tornado.websocket.WebSocketHandler):
         """
         StateHandler.clients.remove(self)
         StateHandler.app_clients.discard(self)
-        if not StateHandler.app_clients:
-            _write_message({"type": "close"}, session=True)
+
+        async def close_wait():
+            await asyncio.sleep(_DISCONNECT_TIMEOUT)
+            if not StateHandler.app_clients:
+                _write_message({"type": "close"}, session=True)
+
+        tornado.ioloop.IOLoop.current().add_callback(close_wait)
 
     @_catch_errors
     async def on_message(self, message):
Repository: voxel51/fiftyone
Issue: [BUG] Open Images dataset ignores classes requirement

### System information

-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS
-   **FiftyOne installed from (pip or source)**: source
-   **FiftyOne version (run `fiftyone --version`)**: 0.11.2.1
-   **Python version**: 3.6.8

### Describe the problem

In the example below, `dataset` should only contain samples with houses, but it contains all downloaded samples:

```py
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

# Download some samples
foz.load_zoo_dataset(
    "open-images-v6",
    split="validation",
    max_samples=50,
)

# This should only contain samples with houses, but actually contains all downloaded samples
dataset = foz.load_zoo_dataset(
    "open-images-v6",
    split="validation",
    classes=["House"],
)

# Verify that some non-House samples exist
no_houses = dataset.match(F("detections.detections").filter(F("label") == "House").length() == 0)
print(len(no_houses))  # should be zero, but isn't
```

Changed files:
fiftyone/utils/openimages.py

diff --git a/fiftyone/utils/openimages.py b/fiftyone/utils/openimages.py
index f06261e52..1f632a324 100644
--- a/fiftyone/utils/openimages.py
+++ b/fiftyone/utils/openimages.py
@@ -311,7 +311,12 @@ class OpenImagesV6DatasetImporter(foud.LabeledImageDatasetImporter):
             valid_ids = all_label_ids + not_all_ids + extra_ids
             valid_ids = valid_ids[:max_samples]
         else:
-            valid_ids = sorted(image_ids)
+            if self.classes is None and self.attrs is None:
+                # No requirements were provided, so always make all samples
+                # available
+                valid_ids = sorted(image_ids)
+            else:
+                valid_ids = sorted(any_label_ids)
 
             if shuffle:
                 random.shuffle(valid_ids)
Repository: voxel51/fiftyone
Issue: [BUG] App grid view does not contain all samples

On `develop`, when I load the quickstart dataset in the App and start scrolling, the grid view exhausts before it gets to the 200th sample:

```py
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)

# Scroll to the end of the grid

# Now show the actual last sample, which differs from above
session.view = dataset.skip(len(dataset) - 1)
```


Changed files:
fiftyone/server/main.py

diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
index f248ee599..5f69ca634 100644
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -254,9 +254,7 @@ class PageHandler(tornado.web.RequestHandler):
 
         samples = await foo.aggregate(
             StateHandler.sample_collection(),
-            view.skip((page - 1) * page_length)._pipeline(
-                attach_frames=True, detach_frames=False
-            ),
+            view._pipeline(attach_frames=True, detach_frames=False),
         ).to_list(page_length + 1)
         convert(samples)
Repository: voxel51/fiftyone
Issue: [BUG] Labelbox Import Uses Incorrect Labelbox ID?

### System information

-   **OS Platform and Distribution**: Linux Ubuntu 20.04
-   **FiftyOne installed from (pip or source)**: pip
-   **FiftyOne version (run `fiftyone --version`)**:  v0.9.1
-   **Python version**: 3.6.9

### Commands to reproduce

We used upload_media_to_labelbox() to upload images from Voxel51 to Labelbox and then performed bounding box labeling. We then exported a JSON file from Labelbox. We ran the import_from_labelbox() function and encountered this error message for each labeled sample:

```
Skipping labels for unknown Labelbox ID 'ckou3botb2y0d0yw47ftmggdr'; provide a `download_dir` if you wish to download media and create samples for new media
```

We were able to overcome this problem by using the "DataRow ID" field instead of the "ID" field from the exported JSON file.

We believe this problem results from the upload_media function, which uses the Labelbox SDK create_data_row() function, which creates a data row object and then the ID of the data row is saved into Voxel51. The ID in the exported JSON file refers to the ID of the label, not the data row.
https://github.com/voxel51/fiftyone/blob/496e402db4eb2974acb12d26f4ffe926ced2d7b0/fiftyone/utils/labelbox.py#L393-L396

### Recommended Fix

We used the following fix and it worked for us.

We changed the "ID" field below:

https://github.com/voxel51/fiftyone/blob/496e402db4eb2974acb12d26f4ffe926ced2d7b0/fiftyone/utils/labelbox.py#L131

to "DataRow ID"


Changed files:
fiftyone/utils/labelbox.py

diff --git a/fiftyone/utils/labelbox.py b/fiftyone/utils/labelbox.py
index 9a1c7de13..32f8d5a15 100644
--- a/fiftyone/utils/labelbox.py
+++ b/fiftyone/utils/labelbox.py
@@ -61,7 +61,7 @@ def import_from_labelbox(
 
         [
             {
-                "ID": <labelbox-id>,
+                "DataRow ID": <labelbox-id>,
                 "Labeled Data": <url-or-None>,
                 "Label": {...}
             }
@@ -128,7 +128,7 @@ def import_from_labelbox(
     # ref: https://github.com/Labelbox/labelbox/blob/7c79b76310fa867dd38077e83a0852a259564da1/exporters/coco-exporter/coco_exporter.py#L33
     with fou.ProgressBar() as pb:
         for d in pb(d_list):
-            labelbox_id = d["ID"]
+            labelbox_id = d["DataRow ID"]
 
             if labelbox_id in id_map:
                 # Get existing sample
@@ -482,7 +482,7 @@ def convert_labelbox_export_to_import(inpath, outpath=None, video_outdir=None):
             )
 
             dout_map[uuid] = {
-                "ID": uuid,
+                "DataRow ID": uuid,
                 "Labeled Data": None,
                 "Label": {"frames": frames_outpath},
             }
@@ -490,7 +490,7 @@ def convert_labelbox_export_to_import(inpath, outpath=None, video_outdir=None):
 
         if uuid not in dout_map:
             dout_map[uuid] = {
-                "ID": uuid,
+                "DataRow ID": uuid,
                 "Labeled Data": None,
                 "Label": {"objects": [], "classifications": []},
             }
Repository: voxel51/fiftyone
Issue: [BUG] Force reencode flag not working

```
from fiftyone.utils import video as fov
<creating the dataset>
fov.reencode_videos(dataset, force_reencode=False)
``` 
does not ignore existing mp4 videos per the documentation, producing unwanted copies of videos.

### System information

-   **Ubuntu 18.04**
-   **source**
-   **0.14**
-   **3.8.11**
-  **ffmpeg version 4.2.4-1ubuntu0.1**

Changed files:
fiftyone/utils/video.py

diff --git a/fiftyone/utils/video.py b/fiftyone/utils/video.py
index e62e46894..fba1a5d86 100644
--- a/fiftyone/utils/video.py
+++ b/fiftyone/utils/video.py
@@ -561,9 +561,12 @@ def _transform_videos(
                 # that all frames exist
                 if not force_reencode and os.path.exists(outpath % 1):
                     continue
-
             elif reencode:
-                outpath = os.path.splitext(inpath)[0] + ".mp4"
+                root, ext = os.path.splitext(inpath)
+                if ext.lower() != ".mp4":
+                    outpath = root + ".mp4"
+                else:
+                    outpath = inpath
             else:
                 outpath = inpath
 
@@ -660,11 +663,10 @@ def _transform_video(
             kwargs["out_opts"] = []
 
     should_reencode = (
-        fps is not None
+        force_reencode
+        or fps is not None
         or size is not None
-        or reencode
-        or in_ext != out_ext
-        or force_reencode
+        or in_ext.lower() != out_ext.lower()
     )
 
     if (inpath == outpath) and should_reencode:
@@ -681,7 +683,6 @@ def _transform_video(
     elif should_reencode:
         with etav.FFmpeg(fps=fps, size=size, **kwargs) as ffmpeg:
             ffmpeg.run(inpath, outpath, verbose=verbose)
-
     elif diff_path:
         etau.copy_file(inpath, outpath)
Repository: voxel51/fiftyone
Issue: [BUG] Image paths not ending with an extension make app fail to open

### System information

-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux
-   **FiftyOne installed from (pip or source)**: pip
-   **FiftyOne version (run `fiftyone --version`)**: 0.14.0
-   **Python version**: 3.8

### Describe the problem

If images in a dataset contain path which do not end with an extension e.g. `/path/to/image` instead of `/path/to/image.jpg`, when launching the fiftyone app, it does not load and shows a 500 error without any clear error message.

I can provide a piece of code to reproduce if really needed but I think my description is fairly straightforward.

### What areas of FiftyOne does this bug affect?

-   [x] `App`: FiftyOne application issue
-   [ ] `Core`: Core `fiftyone` Python library issue
-   [ ] `Server`: Fiftyone server issue

### Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another
member of your organization be willing to contribute a fix for this bug to the
FiftyOne codebase?

-   [ ] Yes. I can contribute a fix for this bug independently.
-   [ ] Yes. I would be willing to contribute a fix for this bug with guidance
        from the FiftyOne community.
-   [x] No. I cannot contribute a bug fix at this time.

Changed files:
fiftyone/server/utils.py

diff --git a/fiftyone/server/utils.py b/fiftyone/server/utils.py
index 85d453097..0925035da 100644
--- a/fiftyone/server/utils.py
+++ b/fiftyone/server/utils.py
@@ -127,7 +127,7 @@ def read_metadata(filepath, metadata=None):
         dict
     """
     mimetype, _ = mimetypes.guess_type(filepath)
-    if mimetype.startswith("video/"):
+    if mimetype and mimetype.startswith("video/"):
         if metadata:
             width = metadata.get("frame_width", None)
             height = metadata.get("frame_height", None)
Repository: voxel51/fiftyone
Issue: [BUG] Unable to clone view whose parent dataset has a brain run

On `fiftyone==0.14.0`, Trying to clone a view into a dataset with a brain run:

```py
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(dataset, brain_key="img_sim")

dataset2 = dataset.limit(1).clone()
```

raises the following error:

```
ValueError: Number of sample IDs (1) does not match number of embeddings (200). You may have missing data/labels that you need to omit from your view
```


Changed files:
fiftyone/core/dataset.py

diff --git a/fiftyone/core/dataset.py b/fiftyone/core/dataset.py
index 334cccb0c..d08d2cbe0 100644
--- a/fiftyone/core/dataset.py
+++ b/fiftyone/core/dataset.py
@@ -4715,8 +4715,8 @@ def _clone_dataset_or_view(dataset_or_view, name):
 
     clone_dataset = load_dataset(name)
 
-    # Clone run results
-    if (
+    # Clone run results (full datasets only)
+    if view is None and (
         dataset.has_annotation_runs
         or dataset.has_brain_runs
         or dataset.has_evaluations
Repository: voxel51/fiftyone
Issue: [BUG] APP non-JavaScript MIME type of "text/plain"

### System information

-   **OS Platform and Distribution**: Windows 10 Pro 10.0.18362.239
-   **FiftyOne installed from (pip or source)**:  pip
-   **FiftyOne version (run `fiftyone --version`)**: 0.9.3
-   **Python version**: both 3.8 | 3.9.2

### Commands to reproduce

```
Following installation steps on https://voxel51.com/docs/fiftyone/getting_started/install.html
pip install fiftyone
```

### Describe the problem

After installing the package (tested installing in both Anaconda environment and venv environment) and running the quickstart example, the app shows only blank screen with errors in console described in Other info / logs section.
The issue does not appear to be bound to specific settings with web browsers on the machine since the same error is given when running the fiftyone as remote on this machine and then accessing it from other machines, but rather by something with running web server as i could not reproduce this by installing fiftyone on other machines.

Any idea what might go wrong ?

### Code to reproduce issue

```
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)
```

### Other info / logs

Google Chrome: `Failed to load module script: The server responded with a non-JavaScript MIME type of "text/plain". Strict MIME type checking is enforced for module scripts per HTML spec.` 
for script 
`http://localhost:5151/_dist_/index.js`

Firefox: `Loading module from “http://localhost:5151/_dist_/index.js” was blocked because of a disallowed MIME type (“text/plain”).`


### What areas of FiftyOne does this bug affect?

-   [x] `App`: FiftyOne application issue
-   [ ] `Core`: Core `fiftyone` Python library issue
-   [ ] `Server`: Fiftyone server issue

### Willingness to contribute

-   [ ] Yes. I can contribute a fix for this bug independently.
-   [x] Yes. I would be willing to contribute a fix for this bug with guidance
        from the FiftyOne community.
-   [ ] No. I cannot contribute a bug fix at this time.


Changed files:
fiftyone/server/main.py

diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
index 738c687dd..d6edc30ed 100644
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -1398,6 +1398,12 @@ class FileHandler(tornado.web.StaticFileHandler):
         self.set_header("content-length", self.get_content_size())
         self.set_header("x-colab-notebook-cache-control", "no-cache")
 
+    def get_content_type(self):
+        if self.absolute_path.endswith(".js"):
+            return "text/javascript"
+
+        return super().get_content_type()
+
 
 class MediaHandler(FileHandler):
     @classmethod
Repository: voxel51/fiftyone
Issue: [BUG] s3_client not defined when downloading Open Images dataset

**OS version**: macOS 10.14.6
**Python version**: 3.8
**FiftyOne version**: 0.9.1

When trying to download + load the Open Images dataset:

```py
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "open-images-v6",
    split="validation",
    label_types=["detections"],
    max_samples=10,
)
```

I encounter the following error:

```
-> 1275     _download_specific_images(valid_ids, split, dataset_dir, num_workers)
   1276 
   1277     if "segmentations" in label_types:
/opt/anaconda3/lib/python3.8/site-packages/fiftyone/utils/openimages.py in _download_specific_images(valid_ids, split, dataset_dir, num_workers, ext)
   1565     with fou.ProgressBar(total=len(inputs)) as pb:
   1566         with multiprocessing.Pool(num_workers) as pool:
-> 1567             for _ in pool.imap_unordered(_do_s3_download, inputs):
   1568                 pb.update()
   1569 
/opt/anaconda3/lib/python3.8/multiprocessing/pool.py in next(self, timeout)
    866         if success:
    867             return value
--> 868         raise value
    869 
    870     __next__ = next                    # XXX
NameError: name 's3_client' is not defined
```

Changed files:
fiftyone/utils/openimages.py

diff --git a/fiftyone/utils/openimages.py b/fiftyone/utils/openimages.py
index 3272abbce..af6ba93ab 100644
--- a/fiftyone/utils/openimages.py
+++ b/fiftyone/utils/openimages.py
@@ -1559,19 +1559,14 @@ def _download_specific_images(
             len(inputs),
         )
 
-    global s3_client
-    s3_client = None
-
     with fou.ProgressBar(total=len(inputs)) as pb:
-        with multiprocessing.Pool(num_workers) as pool:
+        with multiprocessing.Pool(num_workers, _initialize_worker) as pool:
             for _ in pool.imap_unordered(_do_s3_download, inputs):
                 pb.update()
 
 
-def _initialize_pool():
+def _initialize_worker():
     global s3_client
-    if s3_client:
-        return
     s3_client = boto3.client(
         "s3",
         config=botocore.config.Config(signature_version=botocore.UNSIGNED),
@@ -1579,7 +1574,6 @@ def _initialize_pool():
 
 
 def _do_s3_download(args):
-    _initialize_pool()
     filepath, filepath_download = args
     s3_client.download_file(_BUCKET_NAME, filepath_download, filepath)
Repository: voxel51/fiftyone
Issue: Need custom FloatField that supports more numeric types

In this code (taken from https://github.com/voxel51/fiftyone/blob/develop/examples/model_inference/README.md), I had to add `float(confidence)` otherwise I got an error about `confidence`, which was a numpy float32 or something similar, not being a supported value for a `mongoengine.fields.FloatField`. 

```py
for imgs, sample_ids in data_loader:
    predictions, confidences = predict(model, imgs)

    # Add predictions to your FiftyOne dataset
    for sample_id, prediction, confidence in zip(
        sample_ids, predictions, confidences
    ):
        sample = dataset[sample_id]
        sample[model_name] = fo.Classification(label=labels_map[prediction])
        sample["confidence"] = float(confidence)  # float() is required here, but shouldn't need to be...
        sample.save()
```

Kind of hard to believe that MongoEngine doesn't handle casting a `np.float32` into a float, but, alas, it seems like our wrapper around `mongoengine.fields.FloatField` will need to override the `validate()` function below to cast non-int types with `float()` as well...

https://github.com/MongoEngine/mongoengine/blob/4275c2d7b791f5910308a4815a1ba39324dee373/mongoengine/fields.py#L377-L411


Changed files:
fiftyone/core/fields.py

diff --git a/fiftyone/core/fields.py b/fiftyone/core/fields.py
index f74ae8829..809a63586 100644
--- a/fiftyone/core/fields.py
+++ b/fiftyone/core/fields.py
@@ -34,7 +34,19 @@ class IntField(mongoengine.IntField, Field):
 
 
 class FloatField(mongoengine.FloatField, Field):
-    pass
+    def validate(self, value):
+        try:
+            value = float(value)
+        except OverflowError:
+            self.error("The value is too large to be converted to float")
+        except (TypeError, ValueError):
+            self.error("%s could not be converted to float" % value)
+
+        if self.min_value is not None and value < self.min_value:
+            self.error("Float value is too small")
+
+        if self.max_value is not None and value > self.max_value:
+            self.error("Float value is too large")
 
 
 class StringField(mongoengine.StringField, Field):
Repository: voxel51/fiftyone
Issue: fiftyone fails to import if not installed in a virtual environment

Due to at least https://github.com/voxel51/fiftyone/blob/v0.1.3/fiftyone/constants.py#L32

Changed files:
fiftyone/constants.py

diff --git a/fiftyone/constants.py b/fiftyone/constants.py
index 8343f66e8..2d713634b 100644
--- a/fiftyone/constants.py
+++ b/fiftyone/constants.py
@@ -29,7 +29,6 @@ except ImportError:
 
 # Directories
 FIFTYONE_DIR = os.path.abspath(os.path.dirname(__file__))
-FIFTYONE_ENV = os.environ["VIRTUAL_ENV"]
 FIFTYONE_CONFIG_DIR = os.path.join(os.path.expanduser("~"), ".fiftyone")
 FIFTYONE_CONFIG_PATH = os.path.join(FIFTYONE_CONFIG_DIR, "config.json")
 BASE_DIR = os.path.dirname(FIFTYONE_DIR)
@@ -58,7 +57,6 @@ DB_PATH = os.path.join(FIFTYONE_CONFIG_DIR, "var/lib/mongo")
 DB_BIN_PATH = os.path.join(FIFTYONE_DB_BIN_DIR, "mongod")
 DB_CLIENT_BIN_PATH = os.path.join(FIFTYONE_DB_BIN_DIR, "mongo")
 DB_LOG_PATH = os.path.join(FIFTYONE_CONFIG_DIR, "var/log/mongodb/mongo.log")
-os.environ["PATH"] = os.pathsep.join([FIFTYONE_CONFIG_DIR, os.environ["PATH"]])
 START_DB = [
     DB_BIN_PATH,
     "--dbpath",
Repository: voxel51/fiftyone
Issue: Incosistent behavior when deleting datasets

Why does the code below seemingly work when I copy + paste the whole thing into ipython, when, conversely, I get an error if I run the same commands one-by-one manually with, say, 1 second of spacing between each?

```py
import fiftyone as fo

dataset1 = fo.Dataset("test")
dataset1.persistent = True
dataset1.delete()

# works only if run immediately after delete() ??????
dataset1 = fo.Dataset("test")
```

Changed files:
fiftyone/core/dataset.py

diff --git a/fiftyone/core/dataset.py b/fiftyone/core/dataset.py
index e14771dde..f90861819 100644
--- a/fiftyone/core/dataset.py
+++ b/fiftyone/core/dataset.py
@@ -195,7 +195,12 @@ class Dataset(foc.SampleCollection, metaclass=DatasetSingleton):
         self.remove_sample(sample_id)
 
     def __getattribute__(self, name):
-        if name in ["name", "deleted", "_name", "_deleted"]:
+        if name.startswith("__") or name in [
+            "name",
+            "deleted",
+            "_name",
+            "_deleted",
+        ]:
             return super().__getattribute__(name)
 
         if getattr(self, "_deleted", False):
Repository: voxel51/fiftyone
Issue: [BUG] Can't view video datasets with `metadata` in App

When trying to load a video dataset with `fiftyone.core.labels.VideoMetadata` values in its `metadata` field, I get the following error:

```
Traceback (most recent call last):
  File "main.py", line 108, in wrapper
    return func(self, *args, **kwargs)
  File "main.py", line 316, in on_get_frame_labels
    field_labels = _make_image_labels(k, v, frame_number)
  File "main.py", line 156, in _make_image_labels
    fol.ImageLabel.from_dict(label).to_image_labels(name=name),
AttributeError: 'VideoMetadata' object has no attribute 'to_image_labels'
```

After setting the `metadata` to `None`, I can then visualize the dataset.

Changed files:
fiftyone/server/main.py

diff --git a/fiftyone/server/main.py b/fiftyone/server/main.py
index 747f574a7..1f77b987d 100644
--- a/fiftyone/server/main.py
+++ b/fiftyone/server/main.py
@@ -375,17 +375,27 @@ class StateController(Namespace):
 
             labels.add_frame(frame_labels)
 
+        sample_schema = state.dataset.get_field_schema()
         for frame_number in range(
             1, etav.get_frame_count(sample["filepath"]) + 1
         ):
             frame_labels = etav.VideoFrameLabels(frame_number=frame_number)
             for k, v in sample.items():
-                if isinstance(v, dict) and k != "frames" and "_cls" in v:
-                    field_labels = _make_frame_labels(k, v, frame_number)
-                    for obj in field_labels.objects:
-                        obj.frame_number = frame_number
+                if k not in sample_schema:
+                    continue
 
-                    frame_labels.merge_labels(field_labels)
+                field = sample_schema[k]
+                if not isinstance(field, fof.EmbeddedDocumentField):
+                    continue
+
+                if not issubclass(field.document_type, fol.Label):
+                    continue
+
+                field_labels = _make_frame_labels(k, v, frame_number)
+                for obj in field_labels.objects:
+                    obj.frame_number = frame_number
+
+                frame_labels.merge_labels(field_labels)
 
             labels.add_frame(frame_labels, overwrite=False)
Repository: tensorflow/datasets
Issue: Deprecation Warning

Hello,
On page: [TensorFlow Datasets](https://www.tensorflow.org/datasets/overview), the following line of code under the title [```As batched tf.Tensor (batch_size=-1)```](https://www.tensorflow.org/datasets/overview#as_batched_tftensor_batch_size-1) will generate a **Deprecation Warning**.
```python
image, label = tfds.as_numpy(tfds.load(
    'mnist',
    split='test',
    batch_size=-1,
    as_supervised=True,
))

print(type(image), image.shape)
```

The warning message is:
```
WARNING:tensorflow:From e:\\python39\\lib\\site-packages\\tensorflow_datasets\\core\\dataset_builder.py:622: get_single_element (from tensorflow.python.data.experimental.ops.get_single_element) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.get_single_element()`.
```
Thank you,

Changed files:
tensorflow_datasets/core/tf_compat.py, tensorflow_datasets/core/dataset_builder.py

diff --git a/tensorflow_datasets/core/dataset_builder.py b/tensorflow_datasets/core/dataset_builder.py
index f9f313e62..4f50e9c58 100644
--- a/tensorflow_datasets/core/dataset_builder.py
+++ b/tensorflow_datasets/core/dataset_builder.py
@@ -619,7 +619,7 @@ class DatasetBuilder(registered.RegisteredDataset):
     # allow the user to overwritte it.
 
     if wants_full_dataset:
-      return tf.data.experimental.get_single_element(ds)
+      return tf_compat.get_single_element(ds)
     return ds
 
   def _should_cache_ds(self, split, shuffle_files, read_config):
diff --git a/tensorflow_datasets/core/tf_compat.py b/tensorflow_datasets/core/tf_compat.py
index 8ed433f1f..d489f4b71 100644
--- a/tensorflow_datasets/core/tf_compat.py
+++ b/tensorflow_datasets/core/tf_compat.py
@@ -71,6 +71,15 @@ def is_dataset(ds):
   return isinstance(ds, (tf.data.Dataset, tf.compat.v1.data.Dataset))
 
 
+def get_single_element(ds):
+  """Calls `tf.data.Dataset.get_single_element`."""
+  import tensorflow as tf  # pylint: disable=import-outside-toplevel
+  if hasattr(ds, "get_single_element"):  # tf 2.6 and above
+    return ds.get_single_element()
+  else:
+    return tf.data.experimental.get_single_element(ds)
+
+
 def _make_pathlike_fn(fn, nb_path_arg=1):
   """Wrap the function in a PathLike-compatible function."""
Repository: tensorflow/datasets
Issue: malaria dataset URL is out of date

**Short description**
The malaria dataset cannot be downloaded, because the encoded URL is out-of-date.

**Environment information**
* Operating System: Fedora 34
* Python version: 3.9.7
* `tensorflow-datasets`/`tfds-nightly` version: `tensorflow-datasets` 4.5.2
* `tensorflow`/`tf-nightly` version: `tensorflow` 2.8.0

* Does the issue still exists with the last `tfds-nightly` package (`pip install --upgrade tfds-nightly`) ? yes

**Reproduction instructions**

```python
builder = tfds.builder('malaria', data_dir=os.path.join(".", "data"))
builder.download_and_prepare()
```

**Link to logs**
https://gist.github.com/MaximLippeveld/904dc4cd7f7eaabf96291af5e9199d32

**Expected behavior**
Download malaria dataset.

**Additional context**


Changed files:
tensorflow_datasets/image_classification/malaria.py

diff --git a/tensorflow_datasets/image_classification/malaria.py b/tensorflow_datasets/image_classification/malaria.py
index 0ba3b97fa..78b0e797e 100644
--- a/tensorflow_datasets/image_classification/malaria.py
+++ b/tensorflow_datasets/image_classification/malaria.py
@@ -21,7 +21,7 @@ import os
 import tensorflow as tf
 import tensorflow_datasets.public_api as tfds
 
-_URL = "https://ceb.nlm.nih.gov/proj/malaria/cell_images.zip"
+_URL = "https://data.lhncbc.nlm.nih.gov/public/Malaria/cell_images.zip"
 
 _DESCRIPTION = ("""The Malaria dataset contains a total of 27,558 cell images
 with equal instances of parasitized and uninfected cells from the thin blood
Repository: tensorflow/datasets
Issue: Dataset "definite_pronoun_resolution" cannot be loaded

**Short description**

The link for the dataset "definite_pronoun_resolution" does not work now.(http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt). Seems like we also need to contact the dataset authors.

**Environment information**
* Operating System: Ubuntu 18.04.5 LTS (in Colab)
* Python version: 3.7
* `tfds-nightly` version: tfds-nightly-4.4.0.dev202109290107
* `tensorflow` version: 2.6.0

* Does the issue still exists with the last `tfds-nightly` package (`pip install --upgrade tfds-nightly`) ?

Yes

**Reproduction instructions**

```python
import tensorflow_datasets as tfds
tfds.load("definite_pronoun_resolution")
```

**Link to logs**

```
ConnectionError                           Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    514                 raise SSLError(e, request=request)
    515 
--> 516             raise ConnectionError(e, request=request)
    517 
    518         except ClosedPoolError as e:

ConnectionError: HTTPConnectionPool(host='www.hlt.utdallas.edu', port=80): Max retries exceeded with url: /~vince/data/emnlp12/test.c.txt (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5cfe25d550>: Failed to establish a new connection: [Errno 110] Connection timed out'))
```

**Expected behavior**

The dataset successfully downloads


Changed files:
tensorflow_datasets/text/definite_pronoun_resolution.py

diff --git a/tensorflow_datasets/text/definite_pronoun_resolution.py b/tensorflow_datasets/text/definite_pronoun_resolution.py
index c6d34f946..a5b1eb4f3 100644
--- a/tensorflow_datasets/text/definite_pronoun_resolution.py
+++ b/tensorflow_datasets/text/definite_pronoun_resolution.py
@@ -40,7 +40,7 @@ the fourth line contains the correct antecedent. If the target pronoun appears
 more than once in the sentence, its first occurrence is the one to be resolved.
 """
 
-_DATA_URL_PATTERN = 'http://www.hlt.utdallas.edu/~vince/data/emnlp12/{}.c.txt'
+_DATA_URL_PATTERN = 'https://s3.amazonaws.com/datasets.huggingface.co/definite_pronoun_resolution/{}.c.txt'
 
 
 class DefinitePronounResolution(tfds.core.GeneratorBasedBuilder):