Repository: pytorch/torchcodec
Pull request #564: Add core support for decoding from Python file-like objects
Author: login: scotts
Labels: name: CLA Signed
description: This label is managed by the Meta Open Source bot.
Created: 2025-03-14T19:56:04Z | Merged: 2025-03-27T16:54:24Z

The purpose of this PR is to provide functionality in the core API that allows users to provide a Python [file-like object](https://docs.python.org/3/glossary.html#term-file-like-object) as the video for us to decode. Specifically, we're exposing:
```py
def create_from_file_like(
    file_like: Union[io.RawIOBase, io.BytesIO], seek_mode: Optional[str] = None
) -> torch.Tensor:
```
Everything else in this PR is in direct support of what we need to do to support this new function.

On the kind of file-like object, note that we accept and test both [`io.RawIOBase`](https://docs.python.org/3/library/io.html#io.RawIOBase) and [`io.BytesIO`](https://docs.python.org/3/library/io.html#io.BytesIO). I'm confident we should support `RawIOBase`, as it's unbuffered, byte oriented reading. I'm less sure about `BytesIO`, because it is *buffered* byte oriented reading. The current tests pass, but we're not stressing it much.

On what we had to do to expose this new function:

1. Implement pybind11 functions that don't go through PyTorch C++ custom ops.
2. Split the shared libraries into three: `libtorchcodec_decoderN.so`, `libtorchcodec_custom_opsN.so`, `libtorchcodec_pybind_opsN.so`.
3. Generalize how we handle [AVIOContext](https://www.ffmpeg.org/doxygen/2.1/structAVIOContext.html) objects in the C++ VideoDecoder.

Going over each in turn:

**Why directly use pybind11?**

We're already using PyTorch C++ custom ops for our interface between Python and C++, and they already have some dependence on pybind11, so why do we need to create non-custom op pybind11 functions?

From what I can tell, the custom ops were not really intended for what we want to do, which is call C++ code which will itself make callbacks back up to user-provide Python functions. That is, when the user passes us a file-like object, we want FFmpeg to call the read and seek methods on the Python file-like object for all reading and seeking. I don't think custom ops are designed for that kind of dynamic callbacks to Python code. (Custom ops definitely *can* call Python custom ops, but they need to be registered as such ahead of time. We want to call arbitrary Python functions.) I may be wrong here, and we can explore that in the future.

What's I'm more confident of is that we need to store an actual reference to the Python file-like object in the C++ side. Using pybind11 directly, that's easy: we keep a pointer to a `py::object*`. PyTorch custom ops only accept tensors as arguments. We're already smuggling pointers through tensors in the rest of the core API, but we're going from C++ to Python and back to C++. When we store a pointer from C++ in a tensor to go back up to Python, we know it's the right thing. In this instance, we would want to smuggle a pointer from Python through a tensor to C++ - but Python doesn't have pointers. We may be able to get something that will work most of the time by just asking for `id(file_like)`, but even then, I'm not sure how to reliably turn that into a `py::object*` on the C++ side.

If we just use a pybind11 function, all of this difficulty goes away. The cost is that using a file-like object is definitely not going to be compatible with torch.export and torch.compile, but I'm not sure how to make that known to the PyTorch custom ops. This warrants further investigation.

**Why do we need multiple shared libraries?**

I *think* things are simpler if we have a shared library just for the custom ops and a separate shared library just for the pybind11 ops. That then means we need a third library which holds the actual decoder logic. I was not able to get anything working until I made this division, as I am currently using [`importlib.util.spec_from_file_location()`](https://docs.python.org/3/library/importlib.html#importlib.util.spec_from_file_location) and [`importlib.util.module_from_spec()`](https://docs.python.org/3/library/importlib.html#importlib.util.spec_from_file_location) to load the pybind11 module. We just use `torch.ops.load_library()` for the custom ops; that function has machinery that then exposes available functions as fields of the module.

What I'm currently doing works on Linux, but is failing on Mac, so *something* is wrong. It may be possible we don't need to do the split.

**Generalization of handling AVIOContext**

Custom reading and seeking is done in FFmpeg by setting up an `AVIOContext` object. You call [`avio_alloc_context()`](https://ffmpeg.org/doxygen/2.1/avio_8h.html#a853f5149136a27ffba3207d8520172a5) where you provide a pointer to some external state, and then functions for read, seek and write. Then, during decoding, when FFmpeg needs to get more bytes from the file, it calls the callbacks, providing a pointer to the external state. You're responsible for managing that external state in your callbacks.

We already were using this for when users provided us the entire file as just plain bytes. I generalized this handling into three classes:

1. `AVIOContexHolder` which is a base class that knows how to allocate an `AVIOContext`. It cannot be instantiated directly. The `VideoDecoder` can be instantiated with an `AVIOContextHolder` and it uses it appropriately.
2. `AVIOBytesContext` which is the existing functionality we already had. It derives from `AVIOContextHolder`.
3. `AVIOFileLikeContext` which is the new functionality, and it also derives from `AVIOContextHolder`.


Diff:
diff --git a/.github/workflows/cpp_tests.yaml b/.github/workflows/cpp_tests.yaml
index 6ae56c70f..5e31babc7 100644
--- a/.github/workflows/cpp_tests.yaml
+++ b/.github/workflows/cpp_tests.yaml
@@ -34,12 +34,12 @@ jobs:
           python-version: '3.12'
       - name: Update pip
         run: python -m pip install --upgrade pip
-      - name: Install dependencies
+      - name: Install torch dependencies
         run: |
           python -m pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu
-      - name: Install ffmpeg and pkg-config
+      - name: Install ffmpeg, pkg-config and pybind11
         run: |
-          conda install "ffmpeg=${{ matrix.ffmpeg-version-for-tests }}" pkg-config -c conda-forge
+          conda install "ffmpeg=${{ matrix.ffmpeg-version-for-tests }}" pkg-config pybind11 -c conda-forge
           ffmpeg -version
       - name: Build and run C++ tests
         run: |
diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml
index 89e8401de..60bfbfa2e 100644
--- a/.github/workflows/docs.yaml
+++ b/.github/workflows/docs.yaml
@@ -38,6 +38,7 @@ jobs:
       test-infra-repository: pytorch/test-infra
       test-infra-ref: main
       build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
+      pre-script: packaging/pre_build_script.sh
       post-script: packaging/post_build_script.sh
       smoke-test-script: packaging/fake_smoke_test.py
       package-name: torchcodec
diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml
index 71cc071c8..c156a833c 100644
--- a/.github/workflows/lint.yaml
+++ b/.github/workflows/lint.yaml
@@ -63,7 +63,7 @@ jobs:
       - name: Install dependencies and FFmpeg
         run: |
           python -m pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu
-          conda install "ffmpeg=7.0.1" pkg-config -c conda-forge
+          conda install "ffmpeg=7.0.1" pkg-config pybind11 -c conda-forge
           ffmpeg -version
       - name: Build and install torchcodec
         run: |
diff --git a/.github/workflows/linux_cuda_wheel.yaml b/.github/workflows/linux_cuda_wheel.yaml
index 65b06e933..53b5bfc20 100644
--- a/.github/workflows/linux_cuda_wheel.yaml
+++ b/.github/workflows/linux_cuda_wheel.yaml
@@ -48,6 +48,7 @@ jobs:
       test-infra-repository: pytorch/test-infra
       test-infra-ref: main
       build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
+      pre-script: packaging/pre_build_script.sh
       post-script: packaging/post_build_script.sh
       smoke-test-script: packaging/fake_smoke_test.py
       package-name: torchcodec
diff --git a/.github/workflows/linux_wheel.yaml b/.github/workflows/linux_wheel.yaml
index f5e665f55..cd53219f7 100644
--- a/.github/workflows/linux_wheel.yaml
+++ b/.github/workflows/linux_wheel.yaml
@@ -49,6 +49,7 @@ jobs:
       test-infra-repository: pytorch/test-infra
       test-infra-ref: main
       build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
+      pre-script: packaging/pre_build_script.sh
       post-script: packaging/post_build_script.sh
       smoke-test-script: packaging/fake_smoke_test.py
       package-name: torchcodec
diff --git a/.github/workflows/macos_wheel.yaml b/.github/workflows/macos_wheel.yaml
index b7cc965eb..d9472765c 100644
--- a/.github/workflows/macos_wheel.yaml
+++ b/.github/workflows/macos_wheel.yaml
@@ -49,6 +49,7 @@ jobs:
       test-infra-repository: pytorch/test-infra
       test-infra-ref: main
       build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
+      pre-script: packaging/pre_build_script.sh
       post-script: packaging/post_build_script.sh
       smoke-test-script: packaging/fake_smoke_test.py
       runner-type: macos-m1-stable
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index bc3ec3bb0..d516bc272 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -20,6 +20,7 @@ test locally you will need the following dependencies:
   installation already.
 - cmake
 - pkg-config
+- pybind11
 - FFmpeg
 - PyTorch nightly
 
@@ -29,7 +30,7 @@ Start by installing the **nightly** build of PyTorch following the
 Then, the easiest way to install the rest of the dependencies is to run:
 
 ```bash
-conda install cmake pkg-config ffmpeg -c conda-forge
+conda install cmake pkg-config pbyind11 ffmpeg -c conda-forge
 ```
 
 ### Clone and build
diff --git a/packaging/pre_build_script.sh b/packaging/pre_build_script.sh
new file mode 100644
index 000000000..f22244e9c
--- /dev/null
+++ b/packaging/pre_build_script.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -ex
+
+# We need to install pybind11 because we need its CMake helpers in order to
+# compile correctly on Mac. Pybind11 is actually a C++ header-only library,
+# and PyTorch actually has it included. PyTorch, however, does not have the
+# CMake helpers.
+conda install -y pybind11 -c conda-forge
diff --git a/setup.py b/setup.py
index 9120c7fe0..f16521764 100644
--- a/setup.py
+++ b/setup.py
@@ -68,7 +68,7 @@ def run(self):
         super().run()
 
     def build_extension(self, ext):
-        """Call our CMake build system to build libtorchcodec?.so"""
+        """Call our CMake build system to build libtorchcodec*.so"""
         # Setuptools was designed to build one extension (.so file) at a time,
         # calling this method for each Extension object. We're using a
         # CMake-based build where all our extensions are built together at once.
@@ -136,21 +136,27 @@ def copy_extensions_to_source(self):
         This is called by setuptools at the end of .run() during editable installs.
         """
         self.get_finalized_command("build_py")
-        extension = ""
+        extensions = []
         if sys.platform == "linux":
-            extension = "so"
+            extensions = ["so"]
         elif sys.platform == "darwin":
-            extension = "dylib"
+            # Mac has BOTH .dylib and .so as library extensions. Short version
+            # is that a .dylib is a shared library that can be both dynamically
+            # loaded and depended on by other libraries; a .so can only be a
+            # dynamically loaded module. For more, see:
+            #   https://stackoverflow.com/a/2339910
+            extensions = ["dylib", "so"]
         else:
             raise NotImplementedError(
                 "Platforms other than linux/darwin are not supported yet"
             )
 
-        for so_file in self._install_prefix.glob(f"*.{extension}"):
-            assert "libtorchcodec" in so_file.name
-            destination = Path("src/torchcodec/") / so_file.name
-            print(f"Copying {so_file} to {destination}")
-            self.copy_file(so_file, destination, level=self.verbose)
+        for ext in extensions:
+            for lib_file in self._install_prefix.glob(f"*.{ext}"):
+                assert "libtorchcodec" in lib_file.name
+                destination = Path("src/torchcodec/") / lib_file.name
+                print(f"Copying {lib_file} to {destination}")
+                self.copy_file(lib_file, destination, level=self.verbose)
 
 
 NOT_A_LICENSE_VIOLATION_VAR = "I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION"
diff --git a/src/torchcodec/_internally_replaced_utils.py b/src/torchcodec/_internally_replaced_utils.py
index 0833eb82f..a5a3ffa39 100644
--- a/src/torchcodec/_internally_replaced_utils.py
+++ b/src/torchcodec/_internally_replaced_utils.py
@@ -7,11 +7,12 @@
 import importlib
 import sys
 from pathlib import Path
+from types import ModuleType
 
 
 # Copy pasted from torchvision
 # https://github.com/pytorch/vision/blob/947ae1dc71867f28021d5bc0ff3a19c249236e2a/torchvision/_internally_replaced_utils.py#L25
-def _get_extension_path(lib_name):
+def _get_extension_path(lib_name: str) -> str:
     extension_suffixes = []
     if sys.platform == "linux":
         extension_suffixes = importlib.machinery.EXTENSION_SUFFIXES
@@ -31,6 +32,22 @@ def _get_extension_path(lib_name):
     )
     ext_specs = extfinder.find_spec(lib_name)
     if ext_specs is None:
-        raise ImportError
+        raise ImportError(f"No spec found for {lib_name}")
+
+    if ext_specs.origin is None:
+        raise ImportError(f"Existing spec found for {lib_name} does not have an origin")
 
     return ext_specs.origin
+
+
+def _load_pybind11_module(module_name: str, library_path: str) -> ModuleType:
+    spec = importlib.util.spec_from_file_location(
+        module_name,
+        library_path,
+    )
+    if spec is None:
+        raise ImportError(
+            f"Unable to load spec for module {module_name} from path {library_path}"
+        )
+
+    return importlib.util.module_from_spec(spec)
diff --git a/src/torchcodec/decoders/_core/AVIOBytesContext.cpp b/src/torchcodec/decoders/_core/AVIOBytesContext.cpp
new file mode 100644
index 000000000..0d1e9d413
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOBytesContext.cpp
@@ -0,0 +1,70 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOBytesContext::AVIOBytesContext(const void* data, int64_t dataSize)
+    : dataContext_{static_cast<const uint8_t*>(data), dataSize, 0} {
+  TORCH_CHECK(data != nullptr, "Video data buffer cannot be nullptr!");
+  TORCH_CHECK(dataSize > 0, "Video data size must be positive");
+  createAVIOContext(&read, &seek, &dataContext_);
+}
+
+// The signature of this function is defined by FFMPEG.
+int AVIOBytesContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  TORCH_CHECK(
+      dataContext->current <= dataContext->size,
+      "Tried to read outside of the buffer: current=",
+      dataContext->current,
+      ", size=",
+      dataContext->size);
+
+  int64_t numBytesRead = std::min(
+      static_cast<int64_t>(buf_size), dataContext->size - dataContext->current);
+
+  TORCH_CHECK(
+      numBytesRead >= 0,
+      "Tried to read negative bytes: numBytesRead=",
+      numBytesRead,
+      ", size=",
+      dataContext->size,
+      ", current=",
+      dataContext->current);
+
+  if (numBytesRead == 0) {
+    return AVERROR_EOF;
+  }
+
+  std::memcpy(buf, dataContext->data + dataContext->current, numBytesRead);
+  dataContext->current += numBytesRead;
+  return numBytesRead;
+}
+
+// The signature of this function is defined by FFMPEG.
+int64_t AVIOBytesContext::seek(void* opaque, int64_t offset, int whence) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  int64_t ret = -1;
+
+  switch (whence) {
+    case AVSEEK_SIZE:
+      ret = dataContext->size;
+      break;
+    case SEEK_SET:
+      dataContext->current = offset;
+      ret = offset;
+      break;
+    default:
+      break;
+  }
+
+  return ret;
+}
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/AVIOBytesContext.h b/src/torchcodec/decoders/_core/AVIOBytesContext.h
new file mode 100644
index 000000000..411866dc0
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOBytesContext.h
@@ -0,0 +1,32 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace facebook::torchcodec {
+
+// Enables users to pass in the entire video as bytes. Our read and seek
+// functions then traverse the bytes in memory.
+class AVIOBytesContext : public AVIOContextHolder {
+ public:
+  explicit AVIOBytesContext(const void* data, int64_t dataSize);
+
+ private:
+  struct DataContext {
+    const uint8_t* data;
+    int64_t size;
+    int64_t current;
+  };
+
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  DataContext dataContext_;
+};
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/AVIOContextHolder.cpp b/src/torchcodec/decoders/_core/AVIOContextHolder.cpp
new file mode 100644
index 000000000..1fc4f5ecf
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOContextHolder.cpp
@@ -0,0 +1,50 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+void AVIOContextHolder::createAVIOContext(
+    AVIOReadFunction read,
+    AVIOSeekFunction seek,
+    void* heldData,
+    int bufferSize) {
+  TORCH_CHECK(
+      bufferSize > 0,
+      "Buffer size must be greater than 0; is " + std::to_string(bufferSize));
+  auto buffer = static_cast<uint8_t*>(av_malloc(bufferSize));
+  TORCH_CHECK(
+      buffer != nullptr,
+      "Failed to allocate buffer of size " + std::to_string(bufferSize));
+
+  avioContext_.reset(avio_alloc_context(
+      buffer,
+      bufferSize,
+      0,
+      heldData,
+      read,
+      nullptr, // write function; not supported yet
+      seek));
+
+  if (!avioContext_) {
+    av_freep(&buffer);
+    TORCH_CHECK(false, "Failed to allocate AVIOContext");
+  }
+}
+
+AVIOContextHolder::~AVIOContextHolder() {
+  if (avioContext_) {
+    av_freep(&avioContext_->buffer);
+  }
+}
+
+AVIOContext* AVIOContextHolder::getAVIOContext() {
+  return avioContext_.get();
+}
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/AVIOContextHolder.h b/src/torchcodec/decoders/_core/AVIOContextHolder.h
new file mode 100644
index 000000000..26bb06f08
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOContextHolder.h
@@ -0,0 +1,65 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
+
+namespace facebook::torchcodec {
+
+// The AVIOContextHolder serves several purposes:
+//
+//   1. It is a smart pointer for the AVIOContext. It has the logic to create
+//      a new AVIOContext and will appropriately free the AVIOContext when it
+//      goes out of scope. Note that this requires more than just having a
+//      UniqueAVIOContext, as the AVIOContext points to a buffer which must be
+//      freed.
+//   2. It is a base class for AVIOContext specializations. When specializing a
+//      AVIOContext, we need to provide four things:
+//        1. A read callback function.
+//        2. A seek callback function.
+//        3. A write callback function. (Not supported yet; it's for encoding.)
+//        4. A pointer to some context object that has the same lifetime as the
+//           AVIOContext itself. This context object holds the custom state that
+//           tracks the custom behavior of reading, seeking and writing. It is
+//           provided upon AVIOContext creation and to the read, seek and
+//           write callback functions.
+//      While it's not required, it is natural for the derived classes to make
+//      all of the above members. Base classes need to call
+//      createAVIOContext(), ideally in their constructor.
+//  3. A generic handle for those that just need to manage having access to an
+//     AVIOContext, but aren't necessarily concerned with how it was customized:
+//     typically, the VideoDecoder.
+class AVIOContextHolder {
+ public:
+  virtual ~AVIOContextHolder();
+  AVIOContext* getAVIOContext();
+
+ protected:
+  // Make constructor protected to prevent anyone from constructing
+  // an AVIOContextHolder without deriving it. (Ordinarily this would be
+  // enforced by having a pure virtual methods, but we don't have any.)
+  AVIOContextHolder() = default;
+
+  // These signatures are defined by FFmpeg.
+  using AVIOReadFunction = int (*)(void*, uint8_t*, int);
+  using AVIOSeekFunction = int64_t (*)(void*, int64_t, int);
+
+  // Deriving classes should call this function in their constructor.
+  void createAVIOContext(
+      AVIOReadFunction read,
+      AVIOSeekFunction seek,
+      void* heldData,
+      int bufferSize = defaultBufferSize);
+
+ private:
+  UniqueAVIOContext avioContext_;
+
+  // Defaults to 64 KB
+  static const int defaultBufferSize = 64 * 1024;
+};
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp b/src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
new file mode 100644
index 000000000..60d1503ae
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
@@ -0,0 +1,80 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  // Note that we acquire the GIL outside of the loop. This is likely more
+  // efficient than releasing and acquiring it each loop iteration.
+  py::gil_scoped_acquire gil;
+
+  int totalNumRead = 0;
+  while (totalNumRead < buf_size) {
+    int request = buf_size - totalNumRead;
+
+    // The Python method returns the actual bytes, which we access through the
+    // py::bytes wrapper. That wrapper, however, does not provide us access to
+    // the underlying data pointer, which we need for the memcpy below. So we
+    // convert the bytes to a string_view to get access to the data pointer.
+    // Becauase it's a view and not a copy, it should be cheap.
+    auto bytesRead = static_cast<py::bytes>((*fileLike)->attr("read")(request));
+    auto bytesView = static_cast<std::string_view>(bytesRead);
+
+    int numBytesRead = static_cast<int>(bytesView.size());
+    if (numBytesRead == 0) {
+      break;
+    }
+
+    TORCH_CHECK(
+        numBytesRead <= request,
+        "Requested up to ",
+        request,
+        " bytes but, received ",
+        numBytesRead,
+        " bytes. The given object does not conform to read protocol of file object.");
+
+    std::memcpy(buf, bytesView.data(), numBytesRead);
+    buf += numBytesRead;
+    totalNumRead += numBytesRead;
+  }
+
+  return totalNumRead == 0 ? AVERROR_EOF : totalNumRead;
+}
+
+int64_t AVIOFileLikeContext::seek(void* opaque, int64_t offset, int whence) {
+  // We do not know the file size.
+  if (whence == AVSEEK_SIZE) {
+    return AVERROR(EIO);
+  }
+
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+  py::gil_scoped_acquire gil;
+  return py::cast<int64_t>((*fileLike)->attr("seek")(offset, whence));
+}
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/AVIOFileLikeContext.h b/src/torchcodec/decoders/_core/AVIOFileLikeContext.h
new file mode 100644
index 000000000..7be07f2b6
--- /dev/null
+++ b/src/torchcodec/decoders/_core/AVIOFileLikeContext.h
@@ -0,0 +1,54 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// Enables uers to pass in a Python file-like object. We then forward all read
+// and seek calls back up to the methods on the Python object.
+class AVIOFileLikeContext : public AVIOContextHolder {
+ public:
+  explicit AVIOFileLikeContext(py::object fileLike);
+
+ private:
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  // Note that we dynamically allocate the Python object because we need to
+  // strictly control when its destructor is called. We must hold the GIL
+  // when its destructor gets called, as it needs to update the reference
+  // count. It's easiest to control that when it's dynamic memory. Otherwise,
+  // we'd have to ensure whatever enclosing scope holds the object has the GIL,
+  // and that's, at least, hard. For all of the common pitfalls, see:
+  //
+  //   https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors
+  //
+  // We maintain a reference to the file-like object because the file-like
+  // object that was created on the Python side must live as long as our
+  // potential use. That is, even if there are no more references to the object
+  // on the Python side, we require that the object is still live.
+  struct PyObjectDeleter {
+    inline void operator()(py::object* obj) const {
+      if (obj) {
+        py::gil_scoped_acquire gil;
+        delete obj;
+      }
+    }
+  };
+
+  using UniquePyObject = std::unique_ptr<py::object, PyObjectDeleter>;
+  UniquePyObject fileLike_;
+};
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/CMakeLists.txt b/src/torchcodec/decoders/_core/CMakeLists.txt
index 8e4e6c5df..f0a8568fe 100644
--- a/src/torchcodec/decoders/_core/CMakeLists.txt
+++ b/src/torchcodec/decoders/_core/CMakeLists.txt
@@ -3,62 +3,167 @@ project(TorchCodec)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+set(PYBIND11_FINDPYTHON ON)
+find_package(pybind11 REQUIRED)
 find_package(Torch REQUIRED)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
+
+function(make_torchcodec_sublibrary
+    library_name
+    type
+    sources
+    library_dependencies)
+
+    add_library(${library_name} ${type} ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
+        PRIVATE
+        ./../../../../
+        "${TORCH_INSTALL_PREFIX}/include"
+        ${Python3_INCLUDE_DIRS}
+    )
+
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
+    target_link_libraries(
+        ${library_name}
+        PUBLIC
+        ${library_dependencies}
+    )
+endfunction()
+
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target)
+
+    # We create three shared libraries per version of FFmpeg, where the version
+    # is denoted by N:
+    #
+    # 1. libtorchcodec_decoderN.{ext}: Base library which contains the
+    #    implementation of VideoDecoder and everything VideoDecoder needs. On
+    #    Linux, {ext} is so. On Mac, it is dylib.
+    #
+    # 2. libtorchcodec_custom_opsN.{ext}: Implementation of the PyTorch custom
+    #    ops. Depends on libtorchcodec_decoderN.{ext}. On Linux, {ext} is so.
+    #    On Mac, it is dylib.
+    #
+    # 3. libtorchcodec_pybind_opsN.{ext}: Implementation of the pybind11 ops. We
+    #    keep these separate from the PyTorch custom ops because we have to
+    #    load these libraries separately on the Python side. Depends on
+    #    libtorchcodec_decoderN.{ext}. On BOTH Linux and Mac {ext} is so.
+
+    # 1. Create libtorchcodec_decoderN.{ext}.
+    set(decoder_library_name "libtorchcodec_decoder${ffmpeg_major_version}")
+    set(decoder_sources
+        AVIOContextHolder.cpp
         FFMPEGCommon.cpp
-        VideoDecoder.h
         VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
     )
+
     if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
+        list(APPEND decoder_sources CudaDevice.cpp)
     else()
-        list(APPEND sources CPUOnlyDevice.cpp)
+        list(APPEND decoder_sources CPUOnlyDevice.cpp)
     endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
 
-    target_include_directories(
-        ${library_name}
-        PRIVATE
-        ./../../../../
-        "${TORCH_INSTALL_PREFIX}/include"
-        ${Python3_INCLUDE_DIRS}
+    set(decoder_library_dependencies
+        ${ffmpeg_target}
+        ${TORCH_LIBRARIES}
     )
 
-    set(NEEDED_LIBRARIES ${ffmpeg_target} ${TORCH_LIBRARIES}
-        ${Python3_LIBRARIES})
     if(ENABLE_CUDA)
-        list(APPEND NEEDED_LIBRARIES
-            ${CUDA_nppi_LIBRARY} ${CUDA_nppicc_LIBRARY} )
+        list(APPEND decoder_library_dependencies
+            ${CUDA_nppi_LIBRARY}
+            ${CUDA_nppicc_LIBRARY}
+        )
     endif()
-    target_link_libraries(
-        ${library_name}
+
+    make_torchcodec_sublibrary(
+        "${decoder_library_name}"
+        SHARED
+        "${decoder_sources}"
+        "${decoder_library_dependencies}"
+    )
+
+    # 2. Create libtorchcodec_custom_opsN.{ext}.
+    set(custom_ops_library_name "libtorchcodec_custom_ops${ffmpeg_major_version}")
+    set(custom_ops_sources
+        AVIOBytesContext.cpp
+        VideoDecoderOps.cpp
+    )
+    set(custom_ops_dependencies
+        ${decoder_library_name}
+        ${Python3_LIBRARIES}
+    )
+    make_torchcodec_sublibrary(
+        "${custom_ops_library_name}"
+        SHARED
+        "${custom_ops_sources}"
+        "${custom_ops_dependencies}"
+    )
+
+    # 3. Create libtorchcodec_pybind_opsN.so.
+    set(pybind_ops_library_name "libtorchcodec_pybind_ops${ffmpeg_major_version}")
+    set(pybind_ops_sources
+        AVIOFileLikeContext.cpp
+        PyBindOps.cpp
+    )
+    set(pybind_ops_dependencies
+       ${decoder_library_name}
+       pybind11::module # This library dependency makes sure we have the right
+                        # Python libraries included as well as all of the right
+                        # settings so that we can successfully load the shared
+                        # library as a Python module on Mac. If we instead use
+                        # ${Python3_LIBRARIES}, it works on Linux but not on
+                        # Mac.
+    )
+    make_torchcodec_sublibrary(
+        "${pybind_ops_library_name}"
+        MODULE # Note that this not SHARED; otherwise we build the wrong kind
+               # of library on Mac. On Mac, SHARED becomes .dylib and MODULE becomes
+               # a .so. We want pybind11 libraries to become .so. If this is
+               # changed to SHARED, we will be able to succesfully compile a
+               # .dylib, but we will not be able to succesfully import that as
+               # a Python module on Mac.
+        "${pybind_ops_sources}"
+        "${pybind_ops_dependencies}"
+    )
+    # pybind11 limits the visibility of symbols in the shared library to prevent
+    # stray initialization of py::objects. The rest of the object code must
+    # match. See:
+    #   https://pybind11.readthedocs.io/en/stable/faq.html#someclass-declared-with-greater-visibility-than-the-type-of-its-field-someclass-member-wattributes
+    target_compile_options(
+        ${pybind_ops_library_name}
+        PUBLIC
+      "-fvisibility=hidden"
+    )
+    # If we don't make sure this flag is set, we run into segfauls at import
+    # time on Mac. See:
+    #    https://github.com/pybind/pybind11/issues/3907#issuecomment-1170412764
+    target_link_options(
+        ${pybind_ops_library_name}
         PUBLIC
-        ${NEEDED_LIBRARIES}
+        "-undefined dynamic_lookup"
     )
 
-    # We already set the library_name to be libtorchcodecN, so we don't want
-    # cmake to add another "lib" prefix. We do it this way because it makes it
-    # easier to find references to libtorchcodec in the code (e.g. via `git
-    # grep`)
-    set_target_properties(${library_name} PROPERTIES PREFIX "")
+    # Install all libraries.
+    set(
+        all_libraries
+        ${decoder_library_name}
+        ${custom_ops_library_name}
+        ${pybind_ops_library_name}
+    )
 
     # The install step is invoked within CMakeBuild.build_library() in
-    # setup.py and just copies the built .so files from the temp
+    # setup.py and just copies the built files from the temp
     # cmake/setuptools build folder into the CMAKE_INSTALL_PREFIX folder. We
     # still need to manually pass "DESTINATION ..." for cmake to copy those
     # files in CMAKE_INSTALL_PREFIX instead of CMAKE_INSTALL_PREFIX/lib.
     install(
-        TARGETS ${library_name}
+        TARGETS ${all_libraries}
         LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX}
     )
 endfunction()
@@ -76,12 +181,10 @@ if(DEFINED ENV{BUILD_AGAINST_ALL_FFMPEG_FROM_S3})
         ${CMAKE_CURRENT_SOURCE_DIR}/fetch_and_expose_non_gpl_ffmpeg_libs.cmake
     )
 
-
-	make_torchcodec_library(libtorchcodec7 ffmpeg7)
-	make_torchcodec_library(libtorchcodec6 ffmpeg6)
-	make_torchcodec_library(libtorchcodec5 ffmpeg5)
-	make_torchcodec_library(libtorchcodec4 ffmpeg4)
-
+    make_torchcodec_libraries(7 ffmpeg7)
+    make_torchcodec_libraries(6 ffmpeg6)
+    make_torchcodec_libraries(4 ffmpeg4)
+    make_torchcodec_libraries(5 ffmpeg5)
 else()
     message(
         STATUS
@@ -121,10 +224,12 @@ else()
         )
     endif()
 
-    set(libtorchcodec_target_name libtorchcodec${ffmpeg_major_version})
-    # Make libtorchcodec_target_name available in the parent's scope, for the
-    # test's CMakeLists.txt
-    set(libtorchcodec_target_name ${libtorchcodec_target_name} PARENT_SCOPE)
+    make_torchcodec_libraries(${ffmpeg_major_version} PkgConfig::LIBAV)
 
-    make_torchcodec_library(${libtorchcodec_target_name} PkgConfig::LIBAV)
+    # Expose these values updwards so that the test compilation does not need
+    # to re-figure it out. FIXME: it's not great that we just copy-paste the
+    # library names.
+    set(libtorchcodec_library_name "libtorchcodec_decoder${ffmpeg_major_version}" PARENT_SCOPE)
+    set(libtorchcodec_custom_ops_name "libtorchcodec_custom_ops${ffmpeg_major_version}" PARENT_SCOPE)
+    set(libav_include_dirs ${LIBAV_INCLUDE_DIRS} PARENT_SCOPE)
 endif()
diff --git a/src/torchcodec/decoders/_core/FFMPEGCommon.cpp b/src/torchcodec/decoders/_core/FFMPEGCommon.cpp
index 1e3a1421f..8bb41a85d 100644
--- a/src/torchcodec/decoders/_core/FFMPEGCommon.cpp
+++ b/src/torchcodec/decoders/_core/FFMPEGCommon.cpp
@@ -126,88 +126,4 @@ SwrContext* allocateSwrContext(
   return swrContext;
 }
 
-AVIOBytesContext::AVIOBytesContext(
-    const void* data,
-    size_t dataSize,
-    size_t bufferSize)
-    : bufferData_{static_cast<const uint8_t*>(data), dataSize, 0} {
-  auto buffer = static_cast<uint8_t*>(av_malloc(bufferSize));
-  TORCH_CHECK(
-      buffer != nullptr,
-      "Failed to allocate buffer of size " + std::to_string(bufferSize));
-
-  avioContext_.reset(avio_alloc_context(
-      buffer,
-      bufferSize,
-      0,
-      &bufferData_,
-      &AVIOBytesContext::read,
-      nullptr,
-      &AVIOBytesContext::seek));
-
-  if (!avioContext_) {
-    av_freep(&buffer);
-    TORCH_CHECK(false, "Failed to allocate AVIOContext");
-  }
-}
-
-AVIOBytesContext::~AVIOBytesContext() {
-  if (avioContext_) {
-    av_freep(&avioContext_->buffer);
-  }
-}
-
-AVIOContext* AVIOBytesContext::getAVIO() {
-  return avioContext_.get();
-}
-
-// The signature of this function is defined by FFMPEG.
-int AVIOBytesContext::read(void* opaque, uint8_t* buf, int buf_size) {
-  auto bufferData = static_cast<AVIOBufferData*>(opaque);
-  TORCH_CHECK(
-      bufferData->current <= bufferData->size,
-      "Tried to read outside of the buffer: current=",
-      bufferData->current,
-      ", size=",
-      bufferData->size);
-
-  buf_size =
-      FFMIN(buf_size, static_cast<int>(bufferData->size - bufferData->current));
-  TORCH_CHECK(
-      buf_size >= 0,
-      "Tried to read negative bytes: buf_size=",
-      buf_size,
-      ", size=",
-      bufferData->size,
-      ", current=",
-      bufferData->current);
-
-  if (!buf_size) {
-    return AVERROR_EOF;
-  }
-  memcpy(buf, bufferData->data + bufferData->current, buf_size);
-  bufferData->current += buf_size;
-  return buf_size;
-}
-
-// The signature of this function is defined by FFMPEG.
-int64_t AVIOBytesContext::seek(void* opaque, int64_t offset, int whence) {
-  auto bufferData = static_cast<AVIOBufferData*>(opaque);
-  int64_t ret = -1;
-
-  switch (whence) {
-    case AVSEEK_SIZE:
-      ret = bufferData->size;
-      break;
-    case SEEK_SET:
-      bufferData->current = offset;
-      ret = offset;
-      break;
-    default:
-      break;
-  }
-
-  return ret;
-}
-
 } // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/FFMPEGCommon.h b/src/torchcodec/decoders/_core/FFMPEGCommon.h
index c61ae2876..0309bf933 100644
--- a/src/torchcodec/decoders/_core/FFMPEGCommon.h
+++ b/src/torchcodec/decoders/_core/FFMPEGCommon.h
@@ -157,34 +157,4 @@ SwrContext* allocateSwrContext(
 // Returns true if sws_scale can handle unaligned data.
 bool canSwsScaleHandleUnalignedData();
 
-// A struct that holds state for reading bytes from an IO context.
-// We give this to FFMPEG and it will pass it back to us when it needs to read
-// or seek in the memory buffer.
-struct AVIOBufferData {
-  const uint8_t* data;
-  size_t size;
-  size_t current;
-};
-
-// A class that can be used as AVFormatContext's IO context. It reads from a
-// memory buffer that is passed in.
-class AVIOBytesContext {
- public:
-  AVIOBytesContext(const void* data, size_t dataSize, size_t bufferSize);
-  ~AVIOBytesContext();
-
-  // Returns the AVIOContext that can be passed to FFMPEG.
-  AVIOContext* getAVIO();
-
-  // The signature of this function is defined by FFMPEG.
-  static int read(void* opaque, uint8_t* buf, int buf_size);
-
-  // The signature of this function is defined by FFMPEG.
-  static int64_t seek(void* opaque, int64_t offset, int whence);
-
- private:
-  UniqueAVIOContext avioContext_;
-  struct AVIOBufferData bufferData_;
-};
-
 } // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/PyBindOps.cpp b/src/torchcodec/decoders/_core/PyBindOps.cpp
new file mode 100644
index 000000000..0b0f6f177
--- /dev/null
+++ b/src/torchcodec/decoders/_core/PyBindOps.cpp
@@ -0,0 +1,45 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+#include <cstdint>
+#include <string>
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include "src/torchcodec/decoders/_core/VideoDecoder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// In principle, this should be able to return a tensor. But when we try that,
+// we run into the bug reported here:
+//
+//   https://github.com/pytorch/pytorch/issues/136664
+//
+// So we instead launder the pointer through an int, and then use a conversion
+// function on the custom ops side to launder that int into a tensor.
+int64_t create_from_file_like(
+    py::object file_like,
+    std::optional<std::string_view> seek_mode) {
+  VideoDecoder::SeekMode realSeek = VideoDecoder::SeekMode::exact;
+  if (seek_mode.has_value()) {
+    realSeek = seekModeFromString(seek_mode.value());
+  }
+
+  auto avioContextHolder = std::make_unique<AVIOFileLikeContext>(file_like);
+
+  VideoDecoder* decoder =
+      new VideoDecoder(std::move(avioContextHolder), realSeek);
+  return reinterpret_cast<int64_t>(decoder);
+}
+
+PYBIND11_MODULE(decoder_core_pybind_ops, m) {
+  m.def("create_from_file_like", &create_from_file_like);
+}
+
+} // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/VideoDecoder.cpp b/src/torchcodec/decoders/_core/VideoDecoder.cpp
index 3d51b8a35..a379cc0b0 100644
--- a/src/torchcodec/decoders/_core/VideoDecoder.cpp
+++ b/src/torchcodec/decoders/_core/VideoDecoder.cpp
@@ -66,15 +66,13 @@ VideoDecoder::VideoDecoder(const std::string& videoFilePath, SeekMode seekMode)
   initializeDecoder();
 }
 
-VideoDecoder::VideoDecoder(const void* data, size_t length, SeekMode seekMode)
-    : seekMode_(seekMode) {
-  TORCH_CHECK(data != nullptr, "Video data buffer cannot be nullptr!");
-
+VideoDecoder::VideoDecoder(
+    std::unique_ptr<AVIOContextHolder> context,
+    SeekMode seekMode)
+    : seekMode_(seekMode), avioContextHolder_(std::move(context)) {
   setFFmpegLogLevel();
 
-  constexpr int bufferSize = 64 * 1024;
-  ioBytesContext_.reset(new AVIOBytesContext(data, length, bufferSize));
-  TORCH_CHECK(ioBytesContext_, "Failed to create AVIOBytesContext");
+  TORCH_CHECK(avioContextHolder_, "Context holder cannot be null");
 
   // Because FFmpeg requires a reference to a pointer in the call to open, we
   // can't use a unique pointer here. Note that means we must call free if open
@@ -82,7 +80,7 @@ VideoDecoder::VideoDecoder(const void* data, size_t length, SeekMode seekMode)
   AVFormatContext* rawContext = avformat_alloc_context();
   TORCH_CHECK(rawContext != nullptr, "Unable to alloc avformat context");
 
-  rawContext->pb = ioBytesContext_->getAVIO();
+  rawContext->pb = avioContextHolder_->getAVIOContext();
   int status = avformat_open_input(&rawContext, nullptr, nullptr, nullptr);
   if (status != 0) {
     avformat_free_context(rawContext);
@@ -2067,4 +2065,14 @@ FrameDims getHeightAndWidthFromOptionsOrAVFrame(
       videoStreamOptions.width.value_or(avFrame->width));
 }
 
+VideoDecoder::SeekMode seekModeFromString(std::string_view seekMode) {
+  if (seekMode == "exact") {
+    return VideoDecoder::SeekMode::exact;
+  } else if (seekMode == "approximate") {
+    return VideoDecoder::SeekMode::approximate;
+  } else {
+    TORCH_CHECK(false, "Invalid seek mode: " + std::string(seekMode));
+  }
+}
+
 } // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/VideoDecoder.h b/src/torchcodec/decoders/_core/VideoDecoder.h
index 4d3e2f2ce..1f8e423be 100644
--- a/src/torchcodec/decoders/_core/VideoDecoder.h
+++ b/src/torchcodec/decoders/_core/VideoDecoder.h
@@ -12,6 +12,7 @@
 #include <ostream>
 #include <string_view>
 
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
 #include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
 
 namespace facebook::torchcodec {
@@ -34,11 +35,12 @@ class VideoDecoder {
       const std::string& videoFilePath,
       SeekMode seekMode = SeekMode::exact);
 
-  // Creates a VideoDecoder from a given buffer of data. Note that the data is
-  // not owned by the VideoDecoder.
+  // Creates a VideoDecoder using the provided AVIOContext inside the
+  // AVIOContextHolder. The AVIOContextHolder is the base class, and the
+  // derived class will have specialized how the custom read, seek and writes
+  // work.
   explicit VideoDecoder(
-      const void* data,
-      size_t length,
+      std::unique_ptr<AVIOContextHolder> context,
       SeekMode seekMode = SeekMode::exact);
 
   // --------------------------------------------------------------------------
@@ -501,7 +503,7 @@ class VideoDecoder {
   // Stores various internal decoding stats.
   DecodeStats decodeStats_;
   // Stores the AVIOContext for the input buffer.
-  std::unique_ptr<AVIOBytesContext> ioBytesContext_;
+  std::unique_ptr<AVIOContextHolder> avioContextHolder_;
   // Whether or not we have already scanned all streams to update the metadata.
   bool scannedAllStreams_ = false;
   // Tracks that we've already been initialized.
@@ -583,4 +585,6 @@ std::ostream& operator<<(
     std::ostream& os,
     const VideoDecoder::DecodeStats& stats);
 
+VideoDecoder::SeekMode seekModeFromString(std::string_view seekMode);
+
 } // namespace facebook::torchcodec
diff --git a/src/torchcodec/decoders/_core/VideoDecoderOps.cpp b/src/torchcodec/decoders/_core/VideoDecoderOps.cpp
index 786d3f327..bd142d70e 100644
--- a/src/torchcodec/decoders/_core/VideoDecoderOps.cpp
+++ b/src/torchcodec/decoders/_core/VideoDecoderOps.cpp
@@ -11,6 +11,7 @@
 #include <string>
 #include "c10/core/SymIntArrayRef.h"
 #include "c10/util/Exception.h"
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
 #include "src/torchcodec/decoders/_core/VideoDecoder.h"
 
 namespace facebook::torchcodec {
@@ -29,6 +30,7 @@ TORCH_LIBRARY(torchcodec_ns, m) {
   m.def("create_from_file(str filename, str? seek_mode=None) -> Tensor");
   m.def(
       "create_from_tensor(Tensor video_tensor, str? seek_mode=None) -> Tensor");
+  m.def("_convert_to_tensor(int decoder_ptr) -> Tensor");
   m.def(
       "_add_video_stream(Tensor(a!) decoder, *, int? width=None, int? height=None, int? num_threads=None, str? dimension_order=None, int? stream_index=None, str? device=None, str? color_conversion_library=None) -> ()");
   m.def(
@@ -63,13 +65,14 @@ TORCH_LIBRARY(torchcodec_ns, m) {
 }
 
 namespace {
+
 at::Tensor wrapDecoderPointerToTensor(
     std::unique_ptr<VideoDecoder> uniqueDecoder) {
   VideoDecoder* decoder = uniqueDecoder.release();
 
   auto deleter = [decoder](void*) { delete decoder; };
   at::Tensor tensor =
-      at::from_blob(decoder, {sizeof(VideoDecoder)}, deleter, {at::kLong});
+      at::from_blob(decoder, {sizeof(VideoDecoder*)}, deleter, {at::kLong});
   auto videoDecoder = static_cast<VideoDecoder*>(tensor.mutable_data_ptr());
   TORCH_CHECK_EQ(videoDecoder, decoder) << "videoDecoder=" << videoDecoder;
   return tensor;
@@ -100,17 +103,6 @@ OpsAudioFramesOutput makeOpsAudioFramesOutput(
       audioFrames.data,
       torch::tensor(audioFrames.ptsSeconds, torch::dtype(torch::kFloat64)));
 }
-
-VideoDecoder::SeekMode seekModeFromString(std::string_view seekMode) {
-  if (seekMode == "exact") {
-    return VideoDecoder::SeekMode::exact;
-  } else if (seekMode == "approximate") {
-    return VideoDecoder::SeekMode::approximate;
-  } else {
-    throw std::runtime_error("Invalid seek mode: " + std::string(seekMode));
-  }
-}
-
 } // namespace
 
 // ==============================
@@ -137,7 +129,10 @@ at::Tensor create_from_tensor(
     at::Tensor video_tensor,
     std::optional<std::string_view> seek_mode) {
   TORCH_CHECK(video_tensor.is_contiguous(), "video_tensor must be contiguous");
-  void* buffer = video_tensor.mutable_data_ptr();
+  TORCH_CHECK(
+      video_tensor.scalar_type() == torch::kUInt8,
+      "video_tensor must be kUInt8");
+  void* data = video_tensor.mutable_data_ptr();
   size_t length = video_tensor.numel();
 
   VideoDecoder::SeekMode realSeek = VideoDecoder::SeekMode::exact;
@@ -145,8 +140,16 @@ at::Tensor create_from_tensor(
     realSeek = seekModeFromString(seek_mode.value());
   }
 
+  auto contextHolder = std::make_unique<AVIOBytesContext>(data, length);
+
   std::unique_ptr<VideoDecoder> uniqueDecoder =
-      std::make_unique<VideoDecoder>(buffer, length, realSeek);
+      std::make_unique<VideoDecoder>(std::move(contextHolder), realSeek);
+  return wrapDecoderPointerToTensor(std::move(uniqueDecoder));
+}
+
+at::Tensor _convert_to_tensor(int64_t decoder_ptr) {
+  auto decoder = reinterpret_cast<VideoDecoder*>(decoder_ptr);
+  std::unique_ptr<VideoDecoder> uniqueDecoder(decoder);
   return wrapDecoderPointerToTensor(std::move(uniqueDecoder));
 }
 
@@ -550,6 +553,7 @@ void scan_all_streams_to_update_metadata(at::Tensor& decoder) {
 TORCH_LIBRARY_IMPL(torchcodec_ns, BackendSelect, m) {
   m.impl("create_from_file", &create_from_file);
   m.impl("create_from_tensor", &create_from_tensor);
+  m.impl("_convert_to_tensor", &_convert_to_tensor);
   m.impl(
       "_get_json_ffmpeg_library_versions", &_get_json_ffmpeg_library_versions);
 }
diff --git a/src/torchcodec/decoders/_core/__init__.py b/src/torchcodec/decoders/_core/__init__.py
index 490e3d834..9de779f60 100644
--- a/src/torchcodec/decoders/_core/__init__.py
+++ b/src/torchcodec/decoders/_core/__init__.py
@@ -20,6 +20,7 @@
     add_video_stream,
     create_from_bytes,
     create_from_file,
+    create_from_file_like,
     create_from_tensor,
     get_ffmpeg_library_versions,
     get_frame_at_index,
diff --git a/src/torchcodec/decoders/_core/ops.py b/src/torchcodec/decoders/_core/ops.py
index e8efa45f2..0f0bdfe25 100644
--- a/src/torchcodec/decoders/_core/ops.py
+++ b/src/torchcodec/decoders/_core/ops.py
@@ -4,41 +4,68 @@
 # This source code is licensed under the BSD-style license found in the
 # LICENSE file in the root directory of this source tree.
 
+import io
 import json
 import warnings
-from typing import List, Optional, Tuple
+from types import ModuleType
+from typing import List, Optional, Tuple, Union
 
 import torch
 from torch.library import get_ctx, register_fake
 
 from torchcodec._internally_replaced_utils import (  # @manual=//pytorch/torchcodec/src:internally_replaced_utils
     _get_extension_path,
+    _load_pybind11_module,
 )
 
+_pybind_ops: Optional[ModuleType] = None
 
-def load_torchcodec_extension():
-    # Successively try to load libtorchcodec7.so, libtorchcodec6.so,
-    # libtorchcodec5.so, and libtorchcodec4.so. Each of these correspond to an
+
+def load_torchcodec_shared_libraries():
+    # Successively try to load libtorchcodec_*7.so, libtorchcodec_*6.so,
+    # libtorchcodec_*5.so, and libtorchcodec_*4.so. Each of these correspond to an
     # ffmpeg major version. This should cover all potential ffmpeg versions
     # installed on the user's machine.
     #
     # On fbcode, _get_extension_path() is overridden and directly points to the
     # correct .so file, so this for-loop succeeds on the first iteration.
+    #
+    # Note that we use two different methods for loading shared libraries:
+    #
+    #   1. torch.ops.load_library(): For PyTorch custom ops and the C++ only
+    #      libraries the custom ops depend on. Loading libraries through PyTorch
+    #      registers the custom ops with PyTorch's runtime and the ops can be
+    #      accessed through torch.ops after loading.
+    #
+    #   2. importlib: For pybind11 modules. We load them dynamically, rather
+    #      than using a plain import statement. A plain import statement only
+    #      works when the module name and file name match exactly. Our shared
+    #      libraries do not meet those conditions.
 
     exceptions = []
+    pybind_ops_module_name = "decoder_core_pybind_ops"
     for ffmpeg_major_version in (7, 6, 5, 4):
-        library_name = f"libtorchcodec{ffmpeg_major_version}"
+        decoder_library_name = f"libtorchcodec_decoder{ffmpeg_major_version}"
+        custom_ops_library_name = f"libtorchcodec_custom_ops{ffmpeg_major_version}"
+        pybind_ops_library_name = f"libtorchcodec_pybind_ops{ffmpeg_major_version}"
         try:
-            torch.ops.load_library(_get_extension_path(library_name))
+            torch.ops.load_library(_get_extension_path(decoder_library_name))
+            torch.ops.load_library(_get_extension_path(custom_ops_library_name))
+
+            pybind_ops_library_path = _get_extension_path(pybind_ops_library_name)
+            global _pybind_ops
+            _pybind_ops = _load_pybind11_module(
+                pybind_ops_module_name, pybind_ops_library_path
+            )
             return
         except Exception as e:
             # TODO: recording and reporting exceptions this way is OK for now as  it's just for debugging,
             # but we should probably handle that via a proper logging mechanism.
-            exceptions.append(e)
+            exceptions.append((ffmpeg_major_version, e))
 
     traceback = (
         "\n[start of libtorchcodec loading traceback]\n"
-        + "\n".join(str(e) for e in exceptions)
+        + "\n".join(f"FFmpeg version {v}: {str(e)}" for v, e in exceptions)
         + "\n[end of libtorchcodec loading traceback]."
     )
     raise RuntimeError(
@@ -56,7 +83,7 @@ def load_torchcodec_extension():
     )
 
 
-load_torchcodec_extension()
+load_torchcodec_shared_libraries()
 
 
 # Note: We use disallow_in_graph because PyTorch does constant propagation of
@@ -67,6 +94,9 @@ def load_torchcodec_extension():
 create_from_tensor = torch._dynamo.disallow_in_graph(
     torch.ops.torchcodec_ns.create_from_tensor.default
 )
+_convert_to_tensor = torch._dynamo.disallow_in_graph(
+    torch.ops.torchcodec_ns._convert_to_tensor.default
+)
 add_video_stream = torch.ops.torchcodec_ns.add_video_stream.default
 _add_video_stream = torch.ops.torchcodec_ns._add_video_stream.default
 add_audio_stream = torch.ops.torchcodec_ns.add_audio_stream.default
@@ -110,6 +140,13 @@ def create_from_bytes(
     return create_from_tensor(buffer, seek_mode)
 
 
+def create_from_file_like(
+    file_like: Union[io.RawIOBase, io.BytesIO], seek_mode: Optional[str] = None
+) -> torch.Tensor:
+    assert _pybind_ops is not None
+    return _convert_to_tensor(_pybind_ops.create_from_file_like(file_like, seek_mode))
+
+
 # ==============================
 # Abstract impl for the operators. Needed by torch.compile.
 # ==============================
@@ -125,6 +162,11 @@ def create_from_tensor_abstract(
     return torch.empty([], dtype=torch.long)
 
 
+@register_fake("torchcodec_ns::_convert_to_tensor")
+def _convert_to_tensor_abstract(decoder_ptr: int) -> torch.Tensor:
+    return torch.empty([], dtype=torch.long)
+
+
 @register_fake("torchcodec_ns::_add_video_stream")
 def _add_video_stream_abstract(
     decoder: torch.Tensor,
diff --git a/test/decoders/CMakeLists.txt b/test/decoders/CMakeLists.txt
index 3350c92c5..126dd2794 100644
--- a/test/decoders/CMakeLists.txt
+++ b/test/decoders/CMakeLists.txt
@@ -22,11 +22,13 @@ add_executable(
 )
 
 target_include_directories(VideoDecoderTest SYSTEM PRIVATE ${TORCH_INCLUDE_DIRS})
+target_include_directories(VideoDecoderTest SYSTEM PRIVATE ${libav_include_dirs})
 target_include_directories(VideoDecoderTest PRIVATE ../../)
 
 target_link_libraries(
   VideoDecoderTest
-  ${libtorchcodec_target_name}
+  ${libtorchcodec_library_name}
+  ${libtorchcodec_custom_ops_name}
   GTest::gtest_main
 )
 
diff --git a/test/decoders/VideoDecoderTest.cpp b/test/decoders/VideoDecoderTest.cpp
index f7747ef6b..dabe70cd0 100644
--- a/test/decoders/VideoDecoderTest.cpp
+++ b/test/decoders/VideoDecoderTest.cpp
@@ -5,6 +5,7 @@
 // LICENSE file in the root directory of this source tree.
 
 #include "src/torchcodec/decoders/_core/VideoDecoder.h"
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
 
 #include <c10/util/Flags.h>
 #include <gtest/gtest.h>
@@ -48,10 +49,12 @@ class VideoDecoderTest : public testing::TestWithParam<bool> {
       std::ifstream input(filepath, std::ios::binary);
       outputStringStream << input.rdbuf();
       content_ = outputStringStream.str();
+
       void* buffer = content_.data();
-      size_t length = outputStringStream.str().length();
+      size_t length = content_.length();
+      auto contextHolder = std::make_unique<AVIOBytesContext>(buffer, length);
       return std::make_unique<VideoDecoder>(
-          buffer, length, VideoDecoder::SeekMode::approximate);
+          std::move(contextHolder), VideoDecoder::SeekMode::approximate);
     } else {
       return std::make_unique<VideoDecoder>(
           filepath, VideoDecoder::SeekMode::approximate);
diff --git a/test/decoders/test_ops.py b/test/decoders/test_ops.py
index 264417c3f..9efb33f35 100644
--- a/test/decoders/test_ops.py
+++ b/test/decoders/test_ops.py
@@ -23,6 +23,7 @@
     add_video_stream,
     create_from_bytes,
     create_from_file,
+    create_from_file_like,
     create_from_tensor,
     get_ffmpeg_library_versions,
     get_frame_at_index,
@@ -345,7 +346,10 @@ def get_frame1_and_frame_time6(decoder):
         assert_frames_equal(frame_time6, reference_frame_time6.to(device))
 
     @pytest.mark.parametrize("device", cpu_and_cuda())
-    @pytest.mark.parametrize("create_from", ("file", "tensor", "bytes"))
+    @pytest.mark.parametrize(
+        "create_from",
+        ("file", "tensor", "bytes", "file_like_rawio", "file_like_bufferedio"),
+    )
     def test_create_decoder(self, create_from, device):
         path = str(NASA_VIDEO.path)
         if create_from == "file":
@@ -354,10 +358,18 @@ def test_create_decoder(self, create_from, device):
             arr = np.fromfile(path, dtype=np.uint8)
             video_tensor = torch.from_numpy(arr)
             decoder = create_from_tensor(video_tensor)
-        else:  # bytes
+        elif create_from == "bytes":
             with open(path, "rb") as f:
                 video_bytes = f.read()
             decoder = create_from_bytes(video_bytes)
+        elif create_from == "file_like_rawio":
+            decoder = create_from_file_like(open(path, mode="rb", buffering=0), "exact")
+        elif create_from == "file_like_bufferedio":
+            decoder = create_from_file_like(
+                open(path, mode="rb", buffering=4096), "exact"
+            )
+        else:
+            raise ValueError("Oops, double check the parametrization of this test!")
 
         add_video_stream(decoder, device=device)
         frame0, _, _ = get_next_frame(decoder)


Reviews:
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T14:14:38Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T14:25:18Z
user: login: NicolasHug
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:09:06Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:36:47Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:45:27Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:50:21Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:51:31Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:27:15Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:29:05Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T01:45:48Z
user: login: NicolasHug
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T11:23:42Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T12:31:48Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T12:35:03Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T12:39:01Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:13:03Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:14:23Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:31:38Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:35:27Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:50:53Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T20:59:41Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T21:02:19Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T00:53:26Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:07:11Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:08:44Z
user: login: NicolasHug
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T11:15:22Z
user: login: NicolasHug
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T11:50:27Z
user: login: NicolasHug
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T11:52:28Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T18:50:06Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T19:06:48Z
user: login: scotts
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T20:01:17Z
user: login: NicolasHug
body: Thanks a lot for the great work @scotts !

Since the PR is growing quite large, I'm happy to go ahead and merge this now if you'd like, and iterate on the tests as follow-ups.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T12:08:42Z
user: login: NicolasHug
body: Latest changes LGTM still
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T11:01:13Z

Review comments:
diff_hunk: @@ -26,19 +31,37 @@ def load_torchcodec_extension():
     # correct .so file, so this for-loop succeeds on the first iteration.
 
     exceptions = []
+    pybind_ops_module_name = "_torchcodec_pybind_ops"
     for ffmpeg_major_version in (7, 6, 5, 4):
-        library_name = f"libtorchcodec{ffmpeg_major_version}"
+        decoder_library_name = f"libtorchcodec_decoder{ffmpeg_major_version}"
+        custom_ops_library_name = f"libtorchcodec_custom_ops{ffmpeg_major_version}"
+        pybind_ops_library_name = f"libtorchcodec_pybind_ops{ffmpeg_major_version}"
         try:
-            torch.ops.load_library(_get_extension_path(library_name))
+            torch.ops.load_library(_get_extension_path(decoder_library_name))
+            torch.ops.load_library(_get_extension_path(custom_ops_library_name))
+
+            pybind_ops_library_path = _get_extension_path(pybind_ops_library_name)
+            torch.ops.load_library(pybind_ops_library_path)
+            spec = importlib.util.spec_from_file_location(
+                pybind_ops_module_name,
+                pybind_ops_library_path,
+            )
+            if spec is None:
+                raise ImportError(
+                    f"Unable to load spec for module {pybind_ops_module_name} from path {pybind_ops_library_path}"
+                )
+
+            global _pybind_ops
+            _pybind_ops = importlib.util.module_from_spec(spec)
path: src/torchcodec/decoders/_core/ops.py
user: login: scotts
body: This does not yet work on Mac. See: https://github.com/pytorch/torchcodec/actions/runs/13924707065/job/38966320796?pr=564

We're unable to load the spec from the file on line 45. When digging around, it seems like we should be using `ctypes.CDLL` to load a shared library, and that's what `torch.ops.load_library()` does under the hood. Hence why I kept that in for the pybind ops.

TorchAudio does both: https://github.com/pytorch/audio/blob/c670ad81fda266b6598aeeef434583eb98197ae8/src/torio/_extension/utils.py#L105

The common case for pybind11 is that you have:

1. A shared library named `foo.so`.
2. A module declared as `foo` in the `PYBIND11_MODULE` macro in C++.
3. `foo.so` lives in a place Python knows to look.

Then `import foo` should just work.
created_at: 2025-03-18T14:14:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,13 +65,14 @@ TORCH_LIBRARY(torchcodec_ns, m) {
 }
 
 namespace {
+
 at::Tensor wrapDecoderPointerToTensor(
     std::unique_ptr<VideoDecoder> uniqueDecoder) {
   VideoDecoder* decoder = uniqueDecoder.release();
 
   auto deleter = [decoder](void*) { delete decoder; };
   at::Tensor tensor =
-      at::from_blob(decoder, {sizeof(VideoDecoder)}, deleter, {at::kLong});
+      at::from_blob(decoder, {sizeof(VideoDecoder*)}, deleter, {at::kLong});
path: src/torchcodec/decoders/_core/VideoDecoderOps.cpp
user: login: scotts
body: Drive-by: we have been allocating enough space for an entire `VideoDecoder` _object_. But we're not storing the entire object, just a pointer.
created_at: 2025-03-18T14:25:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -351,10 +355,18 @@ def test_create_decoder(self, create_from, device):
             arr = np.fromfile(path, dtype=np.uint8)
             video_tensor = torch.from_numpy(arr)
             decoder = create_from_tensor(video_tensor)
-        else:  # bytes
+        elif create_from == "bytes":
             with open(path, "rb") as f:
                 video_bytes = f.read()
             decoder = create_from_bytes(video_bytes)
+        elif create_from == "file_like_rawio":
+            decoder = create_from_file_like(open(path, mode="rb", buffering=0), "exact")
+        elif create_from == "file_like_bufferedio":
+            decoder = create_from_file_like(
+                open(path, mode="rb", buffering=-4096), "exact"
+            )
path: test/decoders/test_ops.py
user: login: NicolasHug
body: Can you describe intent behind `buffering=-4096`? I'm looking at the [`open()` docs](https://docs.python.org/3/library/functions.html#open) but they don't seem to describe what a negative value means?
created_at: 2025-03-18T17:33:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,6 +18,8 @@
     _get_extension_path,
 )
 
+_pybind_ops: Optional[ModuleType] = None
+
 
 def load_torchcodec_extension():
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: super nit: the name and description of this function could be slightly updated
created_at: 2025-03-18T17:40:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,19 +31,37 @@ def load_torchcodec_extension():
     # correct .so file, so this for-loop succeeds on the first iteration.
 
     exceptions = []
+    pybind_ops_module_name = "_torchcodec_pybind_ops"
     for ffmpeg_major_version in (7, 6, 5, 4):
-        library_name = f"libtorchcodec{ffmpeg_major_version}"
+        decoder_library_name = f"libtorchcodec_decoder{ffmpeg_major_version}"
+        custom_ops_library_name = f"libtorchcodec_custom_ops{ffmpeg_major_version}"
+        pybind_ops_library_name = f"libtorchcodec_pybind_ops{ffmpeg_major_version}"
         try:
-            torch.ops.load_library(_get_extension_path(library_name))
+            torch.ops.load_library(_get_extension_path(decoder_library_name))
+            torch.ops.load_library(_get_extension_path(custom_ops_library_name))
+
+            pybind_ops_library_path = _get_extension_path(pybind_ops_library_name)
+            torch.ops.load_library(pybind_ops_library_path)
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: I need to better understand `spec_from_file_location`  and `module_from_spec`, but I'm wondering:

in the main comment you wrote:

> We just use torch.ops.load_library() for the custom ops

yet it looks like we're using `torch.ops.load_library()` for all 3 extensions. Is that redundant with the use of `spec_from_file_location` / `module_from_spec`  ?
created_at: 2025-03-18T17:42:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,6 +158,11 @@ def create_from_tensor_abstract(
     return torch.empty([], dtype=torch.long)
 
 
+@register_fake("torchcodec_ns::_convert_to_tensor")
+def _convert_to_tensor_abstract(decoder_ptr: int) -> torch.Tensor:
+    return torch.empty([], dtype=torch.long)
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: Curious if `_convert_to_tensor` needs to be exposed as a custom op in Python? It seems to only be used within `create_from_file_like()`, so maybe we don't need to expose it in Python and just handle that logic internally within the C++ implementation of `_pybind_ops.create_from_file_like`?
created_at: 2025-03-18T17:46:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+#include <cstdint>
+#include <string>
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include "src/torchcodec/decoders/_core/VideoDecoder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// In principle, this should be able to return a tensor. But when we try that,
+// we run into the bug reported here:
+//
+//   https://github.com/pytorch/pytorch/issues/136664
+//
+// So we instead launder the pointer through an int, and then use a conversion
+// function on the custom ops side to launder that int into a tensor.
path: src/torchcodec/decoders/_core/PyBindOps.cpp
user: login: NicolasHug
body: ~~Do we even need to return a tensor (or an int, as a fallback)? IIRC the reason we have to return tensors with our custom ops was mostly because of a limitation of the Pytorch custom ops plumbing, but since we're just using pybind here, could we simply return a nice `VideoDecoder` object?~~

OK, this still needs to be a tensor (or int fallback) because we'll be calling core APIs from Python e.g. `core.xyz(decoder, ...)` where `decoder` *is* a tensor.
created_at: 2025-03-18T17:59:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+#include <cstdint>
+#include <string>
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include "src/torchcodec/decoders/_core/VideoDecoder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// In principle, this should be able to return a tensor. But when we try that,
+// we run into the bug reported here:
+//
+//   https://github.com/pytorch/pytorch/issues/136664
+//
+// So we instead launder the pointer through an int, and then use a conversion
+// function on the custom ops side to launder that int into a tensor.
+int64_t create_from_file_like(
+    py::object file_like,
+    std::optional<std::string_view> seek_mode) {
+  VideoDecoder::SeekMode realSeek = VideoDecoder::SeekMode::exact;
+  if (seek_mode.has_value()) {
+    realSeek = seekModeFromString(seek_mode.value());
+  }
+
+  auto contextHolder = std::make_unique<AVIOFileLikeContext>(file_like);
path: src/torchcodec/decoders/_core/PyBindOps.cpp
user: login: NicolasHug
body: Nit:

```
find . -type f | xargs sed -i "s/contextHolder/avioContextHolder/g"
```

to make it clear what "context" this is about.
created_at: 2025-03-18T18:01:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOBytesContext::AVIOBytesContext(const void* data, int64_t dataSize)
+    : dataContext_{static_cast<const uint8_t*>(data), dataSize, 0} {
+  TORCH_CHECK(data != nullptr, "Video data buffer cannot be nullptr!");
+  TORCH_CHECK(dataSize > 0, "Video data size must be positive");
+  createAVIOContext(&read, &seek, &dataContext_);
+}
+
+// The signature of this function is defined by FFMPEG.
+int AVIOBytesContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  TORCH_CHECK(
+      dataContext->current <= dataContext->size,
+      "Tried to read outside of the buffer: current=",
+      dataContext->current,
+      ", size=",
+      dataContext->size);
+
+  buf_size = FFMIN(
+      buf_size, static_cast<int>(dataContext->size - dataContext->current));
+  TORCH_CHECK(
+      buf_size >= 0,
+      "Tried to read negative bytes: buf_size=",
+      buf_size,
+      ", size=",
+      dataContext->size,
+      ", current=",
+      dataContext->current);
+
+  if (!buf_size) {
+    return AVERROR_EOF;
+  }
+  memcpy(buf, dataContext->data + dataContext->current, buf_size);
+  dataContext->current += buf_size;
+  return buf_size;
+}
+
+// The signature of this function is defined by FFMPEG.
+int64_t AVIOBytesContext::seek(void* opaque, int64_t offset, int whence) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  int64_t ret = -1;
+
+  switch (whence) {
+    case AVSEEK_SIZE:
+      ret = dataContext->size;
+      break;
+    case SEEK_SET:
+      dataContext->current = offset;
+      ret = offset;
+      break;
+    default:
+      break;
+  }
+
+  return ret;
+}
+
+} // namespace facebook::torchcodec
path: src/torchcodec/decoders/_core/AVIOBytesContext.cpp
user: login: NicolasHug
body: Q: What's the value of having `AVIOContexHolder`, `AVIOBytesContext`, and  `AVIOFileLikeContext` implemented in 3(6) separate files? I guess it helps contain the size of the compiles .so files (e.g. the pybind extension doesn't need the bytes stuff). Or is it mostly just a convention to have one class per file?

Selfishly I think I would find it easier to read in a single file, but I acknowledge this is a very personal nit.
created_at: 2025-03-18T18:09:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,64 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
+
+namespace facebook::torchcodec {
+
+// The AVIOContextHolder serves several purposes:
+//
+//   1. It is a smart pointer for the AVIOContext. It has the logic to create
+//      a new AVIOContext and will appropriately free the AVIOContext when it
+//      goes out of scope. Note that this requires more than just the having a
path: src/torchcodec/decoders/_core/AVIOContextHolder.h
user: login: NicolasHug
body: ```suggestion
//      goes out of scope. Note that this requires more than just having a
```
created_at: 2025-03-18T18:11:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -137,16 +129,24 @@ at::Tensor create_from_tensor(
     at::Tensor video_tensor,
     std::optional<std::string_view> seek_mode) {
   TORCH_CHECK(video_tensor.is_contiguous(), "video_tensor must be contiguous");
-  void* buffer = video_tensor.mutable_data_ptr();
+  void* data = video_tensor.mutable_data_ptr();
path: src/torchcodec/decoders/_core/VideoDecoderOps.cpp
user: login: NicolasHug
body: Nit, we enforce it in Python before calling this op, but for documentation it might be worth asserting that the dtype is kUint8?
created_at: 2025-03-18T18:24:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: I'm struggling to parse

```Cpp
new py::object(fileLike))
```

We're allocating a new `py::object` from an existing `py::object`, which is `fileLike`? What kind of copy is involved here (if any?)
created_at: 2025-03-18T19:00:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: That's quite likely, IIRC when I was working with Cython stuff, acquiring / releasing the GIL wasn't trivially cheap
created_at: 2025-03-18T19:02:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
+    int chunk_len = static_cast<int>(chunk.length());
+    if (chunk_len == 0) {
+      break;
+    }
+    TORCH_CHECK(
+        chunk_len <= request,
+        "Requested up to ",
+        request,
+        " bytes but, received ",
+        chunk_len,
+        " bytes. The given object does not confirm to read protocol of file object.");
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: 
```suggestion
        " bytes. The given object does not conform to read protocol of file object.");
```
created_at: 2025-03-18T19:03:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: Can you help me understand why casting to `string`? Is it for convenience of keeping both the data and the size together?
created_at: 2025-03-18T19:07:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
+    int chunk_len = static_cast<int>(chunk.length());
+    if (chunk_len == 0) {
+      break;
+    }
+    TORCH_CHECK(
+        chunk_len <= request,
+        "Requested up to ",
+        request,
+        " bytes but, received ",
+        chunk_len,
+        " bytes. The given object does not confirm to read protocol of file object.");
+    memcpy(buf, chunk.data(), chunk_len);
+    buf += chunk_len;
+    num_read += chunk_len;
+  }
+  return num_read == 0 ? AVERROR_EOF : num_read;
+}
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: Above: this can come later, but we should definitely add a test on an object that can only return a small amount of bytes at once, so as to stress test the `while` logic above.
created_at: 2025-03-18T19:08:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOBytesContext::AVIOBytesContext(const void* data, int64_t dataSize)
+    : dataContext_{static_cast<const uint8_t*>(data), dataSize, 0} {
+  TORCH_CHECK(data != nullptr, "Video data buffer cannot be nullptr!");
+  TORCH_CHECK(dataSize > 0, "Video data size must be positive");
+  createAVIOContext(&read, &seek, &dataContext_);
+}
+
+// The signature of this function is defined by FFMPEG.
+int AVIOBytesContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  TORCH_CHECK(
+      dataContext->current <= dataContext->size,
+      "Tried to read outside of the buffer: current=",
+      dataContext->current,
+      ", size=",
+      dataContext->size);
+
+  buf_size = FFMIN(
+      buf_size, static_cast<int>(dataContext->size - dataContext->current));
+  TORCH_CHECK(
+      buf_size >= 0,
+      "Tried to read negative bytes: buf_size=",
+      buf_size,
+      ", size=",
+      dataContext->size,
+      ", current=",
+      dataContext->current);
+
+  if (!buf_size) {
+    return AVERROR_EOF;
+  }
+  memcpy(buf, dataContext->data + dataContext->current, buf_size);
+  dataContext->current += buf_size;
+  return buf_size;
+}
+
+// The signature of this function is defined by FFMPEG.
+int64_t AVIOBytesContext::seek(void* opaque, int64_t offset, int whence) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  int64_t ret = -1;
+
+  switch (whence) {
+    case AVSEEK_SIZE:
+      ret = dataContext->size;
+      break;
+    case SEEK_SET:
+      dataContext->current = offset;
+      ret = offset;
+      break;
+    default:
+      break;
+  }
+
+  return ret;
+}
+
+} // namespace facebook::torchcodec
path: src/torchcodec/decoders/_core/AVIOBytesContext.cpp
user: login: scotts
body: I used to have no new files, but I ran into problems with testing. That is, I had `AVIOContextHolder` defined in `FFMPEGCommon.h`, `AVIOBytesContext` defined in `VideoDecoderOps.cpp` and `AVIOFileLikeContext` defined in `PyBindOps.cpp`. That ran into problems with testing in C++, as I needed access to the class declaration for `AVIOBytesContext` in `VideoDecoderTest.cpp`. Then I broke things down this way.

The reason I broke things out in this manner is that it's normal C++ convention to define separate files for classes, and to keep the implementation out of the header. In both instances, I'm trying to limit visibility; that is, the custom ops code doesn't need to know about `AVIOFileLikeContext`, and I think it's easier to reason about things when definitions are limited to only the scope they're needed.

I also pulled `AVIOContextHolder` out of `FFMPEGCommon.h` because it has significant logic unto itself. Right now, `FFMPEGCommon.h` is mostly about thin wrappers around FFmpeg stuff. 

In this breakdown, I think there's some clarify in the fact that `AVIOContextHolder` and the classes that derive it actually don't know about `VideoDecoder`. 
created_at: 2025-03-18T19:36:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: Correct. We're invoking `py::object`'s [copy constructor](https://en.cppreference.com/w/cpp/language/copy_constructor) on `fileLike`. That resulting object is dynamically allocated on the heap, and its pointer is returned by `new`. The pointer to that object is then owned by `AVIOFileLikeContext::fileLike_`.

It may be instructive to contrast this to a stack allocated object, in statements that appear in normal code blocks:
```cpp
py::object copyAllocatedOnStack(fileLike);
py::object copyAlsoAllocatedOnStack{fileLike}; // new style initializer list
py::object* copyAllocatedOnHeap = new py::object(fileLike);
```
created_at: 2025-03-18T19:45:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+#include <cstdint>
+#include <string>
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include "src/torchcodec/decoders/_core/VideoDecoder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// In principle, this should be able to return a tensor. But when we try that,
+// we run into the bug reported here:
+//
+//   https://github.com/pytorch/pytorch/issues/136664
+//
+// So we instead launder the pointer through an int, and then use a conversion
+// function on the custom ops side to launder that int into a tensor.
path: src/torchcodec/decoders/_core/PyBindOps.cpp
user: login: scotts
body: Yes, exactly. I want this functionality to slot into our current APIs, not having a mirror set of APIs.
created_at: 2025-03-18T19:50:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,6 +18,8 @@
     _get_extension_path,
 )
 
+_pybind_ops: Optional[ModuleType] = None
+
 
 def load_torchcodec_extension():
path: src/torchcodec/decoders/_core/ops.py
user: login: scotts
body: I'd love to. :)
created_at: 2025-03-18T19:51:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,6 +158,11 @@ def create_from_tensor_abstract(
     return torch.empty([], dtype=torch.long)
 
 
+@register_fake("torchcodec_ns::_convert_to_tensor")
+def _convert_to_tensor_abstract(decoder_ptr: int) -> torch.Tensor:
+    return torch.empty([], dtype=torch.long)
path: src/torchcodec/decoders/_core/ops.py
user: login: scotts
body: It's needed because we can't handle that logic on the C++ side. That is, pybind11 should be able to just return a tensor. But when we do that, we run into a PyTorch core bug: https://github.com/pytorch/pytorch/issues/136664. In order to avoid that bug, we can't return a tensor from pybind11.

Our work-around, then, is to launder the pointer though an `int` on the pybind11 side, and then use a PyTorch custom op to launder that `int` into a tensor. We need to expose `_convert_to_tensor()` to Python because we put those calls together on the Python side. Notably, we never expose `_convert_to_tensor()` outside of `ops.py`. It's only used to implement `create_from_file_like()`.
created_at: 2025-03-18T20:27:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -351,10 +355,18 @@ def test_create_decoder(self, create_from, device):
             arr = np.fromfile(path, dtype=np.uint8)
             video_tensor = torch.from_numpy(arr)
             decoder = create_from_tensor(video_tensor)
-        else:  # bytes
+        elif create_from == "bytes":
             with open(path, "rb") as f:
                 video_bytes = f.read()
             decoder = create_from_bytes(video_bytes)
+        elif create_from == "file_like_rawio":
+            decoder = create_from_file_like(open(path, mode="rb", buffering=0), "exact")
+        elif create_from == "file_like_bufferedio":
+            decoder = create_from_file_like(
+                open(path, mode="rb", buffering=-4096), "exact"
+            )
path: test/decoders/test_ops.py
user: login: scotts
body: Whoops, I misread the docs. It should be positive.
created_at: 2025-03-18T20:29:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,19 +31,37 @@ def load_torchcodec_extension():
     # correct .so file, so this for-loop succeeds on the first iteration.
 
     exceptions = []
+    pybind_ops_module_name = "_torchcodec_pybind_ops"
     for ffmpeg_major_version in (7, 6, 5, 4):
-        library_name = f"libtorchcodec{ffmpeg_major_version}"
+        decoder_library_name = f"libtorchcodec_decoder{ffmpeg_major_version}"
+        custom_ops_library_name = f"libtorchcodec_custom_ops{ffmpeg_major_version}"
+        pybind_ops_library_name = f"libtorchcodec_pybind_ops{ffmpeg_major_version}"
         try:
-            torch.ops.load_library(_get_extension_path(library_name))
+            torch.ops.load_library(_get_extension_path(decoder_library_name))
+            torch.ops.load_library(_get_extension_path(custom_ops_library_name))
+
+            pybind_ops_library_path = _get_extension_path(pybind_ops_library_name)
+            torch.ops.load_library(pybind_ops_library_path)
path: src/torchcodec/decoders/_core/ops.py
user: login: scotts
body: Good question! I don't know. It's what TorchAudio does:

https://github.com/pytorch/audio/blob/c670ad81fda266b6598aeeef434583eb98197ae8/src/torio/_extension/utils.py#L101-L109

That is, it uses both `torch.ops.load_library()` and functions from `importlib`. I've gotten Linux working without the `torch.ops.load_library()` on the pybind library. What we have here was my last attempt at seeing what might work for Mac.
created_at: 2025-03-19T01:45:48Z
author_association: CONTRIBUTOR
path: test/decoders/test_ops.py
user: login: NicolasHug
body: We can definitely add more tests later especially when we start implementing the public (non-core) APIs, but even for this first PR I think it'd be good to have a small integration test with a custom Python object that exposes `seek()` and `read()`. And we could assert e.g. `num_seek_calls`, `num_read_calls`, etc.
created_at: 2025-03-19T10:06:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: I think what I'm struggling to understand is: what kind of copy is being made within the copy constructor? Is it a shallow, or a deep copy? E.g., if we change the internal state of that py::object in C++ (by e.g. calling seek()), is that change reflected on the original object in Python (shallow copy), or is it just affecting the copy (deep copy)?

I'm not finding a definite answer from the [docs](https://pybind11.readthedocs.io/en/stable/reference.html#_CPPv46object), but I'm also probably missing some context. I suspect that since the py::object is ref-counted, it means it's a "shallow" copy?
created_at: 2025-03-19T10:27:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,49 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// Enables uers to pass in a Python file-like object. We then forward all read
+// and seek calls back up to the methods on the Python object.
+class AVIOFileLikeContext : public AVIOContextHolder {
+ public:
+  explicit AVIOFileLikeContext(py::object fileLike);
+
+ private:
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  // Note that we dynamically allocate the Python object because we need to
+  // strictly control when its destructor is called. We must hold the GIL
+  // when its destructor gets called, as it needs to update the reference
+  // count. It's easiest to control that when it's dynamic memory. Otherwise,
+  // we'd have to ensure whatever enclosing scope holds the object has the GIL,
+  // and that's, at least, hard. For all of the common pitfalls, see:
+  //
+  //   https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors
+  struct PyObjectDeleter {
+    inline void operator()(py::object* obj) const {
+      if (obj) {
+        py::gil_scoped_acquire gil;
+        delete obj;
+      }
+    }
+  };
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.h
user: login: NicolasHug
body: Do you know why the `py::object'`s destructor doesn't hold the GIL? IIUC it's never valid to delete a `py::object` without holding it, so why wouldn't this be done automatically?
created_at: 2025-03-19T10:35:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,64 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
+
+namespace facebook::torchcodec {
+
+// The AVIOContextHolder serves several purposes:
+//
+//   1. It is a smart pointer for the AVIOContext. It has the logic to create
+//      a new AVIOContext and will appropriately free the AVIOContext when it
+//      goes out of scope. Note that this requires more than just having a
+//      UniqueAVIOContext, as the AVIOContext points to a buffer which must be
+//      freed.
+//   2. It is a base class for AVIOContext specializations. When specializing a
+//      AVIOContext, we need to provide four things:
+//        1. A read callback function.
+//        2. A seek callback function.
+//        3. A write callback function. (Not supported yet; it's for encoding.)
+//        4. A pointer to some context object that has the same lifetime as the
+//           AVIOContext itself. This context object holds the custom state that
+//           tracks the custom behavior of reading, seeking and writing. It is
+//           provided upon AVIOContext creation and to the read, seek and
+//           write callback functions.
+//      While it's not required, it is natural for the derived classes to make
+//      all of the above members. Base classes need to call
+//      createAVIOContext(), ideally in their constructor.
+//  3. A generic handle for those that just need to manage having access to an
+//     AVIOContext, but aren't necessarily concerned with how it was customized.
path: src/torchcodec/decoders/_core/AVIOContextHolder.h
user: login: NicolasHug
body: Nit:

```suggestion
//  3. A generic handle for those that just need to manage having access to an
//     AVIOContext, but aren't necessarily concerned with how it was customized:
//     typically, the VideoDecoder.
```
created_at: 2025-03-19T10:38:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,49 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// Enables uers to pass in a Python file-like object. We then forward all read
+// and seek calls back up to the methods on the Python object.
+class AVIOFileLikeContext : public AVIOContextHolder {
+ public:
+  explicit AVIOFileLikeContext(py::object fileLike);
+
+ private:
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  // Note that we dynamically allocate the Python object because we need to
+  // strictly control when its destructor is called. We must hold the GIL
+  // when its destructor gets called, as it needs to update the reference
+  // count. It's easiest to control that when it's dynamic memory. Otherwise,
+  // we'd have to ensure whatever enclosing scope holds the object has the GIL,
+  // and that's, at least, hard. For all of the common pitfalls, see:
+  //
+  //   https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors
+  struct PyObjectDeleter {
+    inline void operator()(py::object* obj) const {
+      if (obj) {
+        py::gil_scoped_acquire gil;
+        delete obj;
+      }
+    }
+  };
+
+  using UniquePyObject = std::unique_ptr<py::object, PyObjectDeleter>;
+  UniquePyObject fileLike_;
+};
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.h
user: login: NicolasHug
body: I was going to ask why we need the fileLike to be ref-counted, (i.e. why not just use `py::handle` instead), and then realized we need the file-like object to be alive for as long as the `VideoDecoder` is alive. Not sure if it's worth documenting?
created_at: 2025-03-19T10:42:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,64 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
+
+namespace facebook::torchcodec {
+
+// The AVIOContextHolder serves several purposes:
+//
+//   1. It is a smart pointer for the AVIOContext. It has the logic to create
+//      a new AVIOContext and will appropriately free the AVIOContext when it
+//      goes out of scope. Note that this requires more than just having a
+//      UniqueAVIOContext, as the AVIOContext points to a buffer which must be
+//      freed.
+//   2. It is a base class for AVIOContext specializations. When specializing a
+//      AVIOContext, we need to provide four things:
+//        1. A read callback function.
+//        2. A seek callback function.
+//        3. A write callback function. (Not supported yet; it's for encoding.)
+//        4. A pointer to some context object that has the same lifetime as the
+//           AVIOContext itself. This context object holds the custom state that
+//           tracks the custom behavior of reading, seeking and writing. It is
+//           provided upon AVIOContext creation and to the read, seek and
+//           write callback functions.
+//      While it's not required, it is natural for the derived classes to make
+//      all of the above members. Base classes need to call
path: src/torchcodec/decoders/_core/AVIOContextHolder.h
user: login: NicolasHug
body: Q: what would be the alternative here, if read/seek weren't members? Functions that take a subclass object as input (as in C-like OOP)?
created_at: 2025-03-19T10:47:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,6 +18,8 @@
     _get_extension_path,
 )
 
+_pybind_ops: Optional[ModuleType] = None
+
 
 def load_torchcodec_extension():
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: Related: we should update `libtorchcodec?.so` to `libtorchcodec*.so` in

https://github.com/pytorch/torchcodec/blob/b83048001d5ae9f3aa7024a091acf78a7be12a0a/setup.py#L70-L71


created_at: 2025-03-19T10:52:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: I could be wrong but it doesn't seem like we need to explicitly add `ffmpeg_include_dirs` anywhere. IIRC, the ffmpeg target we have are smart targets that will automatically add they INCLUDE dirs when calling `target_link_libraries`.

I tried removing those locally and things worked.
created_at: 2025-03-19T11:03:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
 
-    target_include_directories(
-        ${library_name}
+    add_library(${library_name} SHARED ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
         PRIVATE
         ./../../../../
         "${TORCH_INSTALL_PREFIX}/include"
         ${Python3_INCLUDE_DIRS}
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: We probably don't need to worry about this (at least for now), but I suspect the `libtorchcodec_decoder?.so` libraries don't (or shouldn't) need the Python headers?

Similarly the pybind library probably doesn't need the torch headers.
created_at: 2025-03-19T11:06:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,12 +137,10 @@ if(DEFINED ENV{BUILD_AGAINST_ALL_FFMPEG_FROM_S3})
         ${CMAKE_CURRENT_SOURCE_DIR}/fetch_and_expose_non_gpl_ffmpeg_libs.cmake
     )
 
-
-	make_torchcodec_library(libtorchcodec7 ffmpeg7)
-	make_torchcodec_library(libtorchcodec6 ffmpeg6)
-	make_torchcodec_library(libtorchcodec5 ffmpeg5)
-	make_torchcodec_library(libtorchcodec4 ffmpeg4)
-
+    make_torchcodec_libraries(7 ffmpeg7 $ffmpeg7_INCLUDE_DIRs)
+    make_torchcodec_libraries(6 ffmpeg6 $ffmpeg6_INCLUDE_DIRS)
+    make_torchcodec_libraries(4 ffmpeg4 $ffmpeg4_INCLUDE_DIRS)
+    make_torchcodec_libraries(5 ffmpeg5 $ffmpeg5_INCLUDE_DIRS)
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: See comment about `ffmpeg_include_dirs`, I think this could just be:

```suggestion
    make_torchcodec_libraries(7 ffmpeg7 )
    make_torchcodec_libraries(6 ffmpeg6)
    make_torchcodec_libraries(4 ffmpeg4)
    make_torchcodec_libraries(5 ffmpeg5)
```

(same below with ${LIBAV_INCLUDE_DIRS}).
created_at: 2025-03-19T11:18:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: Nit, I could be reading this the wrong way, but for me a "dependent library" is a **downstream** dependency, i.e. a library that depends on `library_name`. Here, we use "dependent library" to just mean "dependency", i.e. a library that `library_name` depends on?
created_at: 2025-03-19T11:19:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
 
-    target_include_directories(
-        ${library_name}
+    add_library(${library_name} SHARED ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
         PRIVATE
         ./../../../../
         "${TORCH_INSTALL_PREFIX}/include"
         ${Python3_INCLUDE_DIRS}
+        ${ffmpeg_include_dirs}
     )
 
-    set(NEEDED_LIBRARIES ${ffmpeg_target} ${TORCH_LIBRARIES}
-        ${Python3_LIBRARIES})
-    if(ENABLE_CUDA)
-        list(APPEND NEEDED_LIBRARIES
-            ${CUDA_nppi_LIBRARY} ${CUDA_nppicc_LIBRARY} )
-    endif()
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
     target_link_libraries(
         ${library_name}
         PUBLIC
-        ${NEEDED_LIBRARIES}
+        ${dependent_libraries}
     )
+endfunction()
 
-    # We already set the library_name to be libtorchcodecN, so we don't want
-    # cmake to add another "lib" prefix. We do it this way because it makes it
-    # easier to find references to libtorchcodec in the code (e.g. via `git
-    # grep`)
-    set_target_properties(${library_name} PROPERTIES PREFIX "")
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target
+    ffmpeg_include_dirs)
+
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: This can certainly be done in another PR but I think it would help to write a big comment to explain what we build, and why we have 3 different extensions. Maybe a link to your main comment on https://github.com/pytorch/torchcodec/pull/564/ is enough.
created_at: 2025-03-19T11:20:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,49 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// Enables uers to pass in a Python file-like object. We then forward all read
+// and seek calls back up to the methods on the Python object.
+class AVIOFileLikeContext : public AVIOContextHolder {
+ public:
+  explicit AVIOFileLikeContext(py::object fileLike);
+
+ private:
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  // Note that we dynamically allocate the Python object because we need to
+  // strictly control when its destructor is called. We must hold the GIL
+  // when its destructor gets called, as it needs to update the reference
+  // count. It's easiest to control that when it's dynamic memory. Otherwise,
+  // we'd have to ensure whatever enclosing scope holds the object has the GIL,
+  // and that's, at least, hard. For all of the common pitfalls, see:
+  //
+  //   https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors
+  struct PyObjectDeleter {
+    inline void operator()(py::object* obj) const {
+      if (obj) {
+        py::gil_scoped_acquire gil;
+        delete obj;
+      }
+    }
+  };
+
+  using UniquePyObject = std::unique_ptr<py::object, PyObjectDeleter>;
+  UniquePyObject fileLike_;
+};
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.h
user: login: scotts
body: Definitely, if it wasn't immediately obvious to you, we should document it. 
created_at: 2025-03-19T12:31:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,49 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+
+#include "src/torchcodec/decoders/_core/AVIOContextHolder.h"
+
+namespace py = pybind11;
+
+namespace facebook::torchcodec {
+
+// Enables uers to pass in a Python file-like object. We then forward all read
+// and seek calls back up to the methods on the Python object.
+class AVIOFileLikeContext : public AVIOContextHolder {
+ public:
+  explicit AVIOFileLikeContext(py::object fileLike);
+
+ private:
+  static int read(void* opaque, uint8_t* buf, int buf_size);
+  static int64_t seek(void* opaque, int64_t offset, int whence);
+
+  // Note that we dynamically allocate the Python object because we need to
+  // strictly control when its destructor is called. We must hold the GIL
+  // when its destructor gets called, as it needs to update the reference
+  // count. It's easiest to control that when it's dynamic memory. Otherwise,
+  // we'd have to ensure whatever enclosing scope holds the object has the GIL,
+  // and that's, at least, hard. For all of the common pitfalls, see:
+  //
+  //   https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors
+  struct PyObjectDeleter {
+    inline void operator()(py::object* obj) const {
+      if (obj) {
+        py::gil_scoped_acquire gil;
+        delete obj;
+      }
+    }
+  };
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.h
user: login: scotts
body: Actually _storing_ a `py::object` on the C++ side is not the common case. More common is that your C++ function receives a `py::object` through a pybind registered function, and you only do this inside of that pybind function. pybind guarantees that the GIL is held when it calls your C++ function.

Note that the PyTorch custom ops do _not_ hold the GIL, also on purpose. That's why we have to be GIL-aware in our code.
created_at: 2025-03-19T12:35:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: In general, it's whatever copy the class being copied decided it is. :)

In this case, `py::object` is effectively a smart pointer back to a Python object. Under-the-hood, it's going to actually hold a pointer to the C struct [`PyObject`](https://docs.python.org/3/c-api/structures.html#c.PyObject) which is the actual implementation of the Python level object. So it's definitely a shallow-copy in the sense of the object itself, as `py::object` is a ref-counted alias to the actual object.
created_at: 2025-03-19T12:39:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,64 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#pragma once
+
+#include "src/torchcodec/decoders/_core/FFMPEGCommon.h"
+
+namespace facebook::torchcodec {
+
+// The AVIOContextHolder serves several purposes:
+//
+//   1. It is a smart pointer for the AVIOContext. It has the logic to create
+//      a new AVIOContext and will appropriately free the AVIOContext when it
+//      goes out of scope. Note that this requires more than just having a
+//      UniqueAVIOContext, as the AVIOContext points to a buffer which must be
+//      freed.
+//   2. It is a base class for AVIOContext specializations. When specializing a
+//      AVIOContext, we need to provide four things:
+//        1. A read callback function.
+//        2. A seek callback function.
+//        3. A write callback function. (Not supported yet; it's for encoding.)
+//        4. A pointer to some context object that has the same lifetime as the
+//           AVIOContext itself. This context object holds the custom state that
+//           tracks the custom behavior of reading, seeking and writing. It is
+//           provided upon AVIOContext creation and to the read, seek and
+//           write callback functions.
+//      While it's not required, it is natural for the derived classes to make
+//      all of the above members. Base classes need to call
path: src/torchcodec/decoders/_core/AVIOContextHolder.h
user: login: scotts
body: The functions could actually just be static functions defined in the .cpp file. They wouldn't have access to the private members of the derived class, although that also can be gotten around through the use of `friend`. As for the the state, technically it would also be defined statically in the .cpp file, but that's harder to get right as you want the state to have the same lifetime as the object itself. So, possible, just very tricky.

The point I'm making here is there is nothing in `AVIOContextHolder` that actually forces you to make the functions and state members of the deriving class. It's just really convenient to do so.
created_at: 2025-03-19T20:13:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: Then let's move it out of the loop.
created_at: 2025-03-19T20:14:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: This is code copied from TorchAudio, and I thought it was necessary for that reason. But looking at the docs, I think we can just call a C++ version of `len()` on the returned `py::bytes`. I'll look into that.
created_at: 2025-03-19T20:31:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: Looking more: it's so that we have a pointer to the data that we can pass to `memcpy()`.
created_at: 2025-03-19T20:35:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: If it's at all confusing, let's change the name. I think `library_dependencies` clears that up.
created_at: 2025-03-19T20:50:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: Interesting, I put them in because I was getting failures. It's also possible I made more changes which made them not necessary again.
created_at: 2025-03-19T20:59:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: Yup! Works for me too. I'm not sure what was going on when I added them. That will simplify the compilation as we don't need to pass as much info around.
created_at: 2025-03-19T21:02:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,58 +7,119 @@ find_package(Torch REQUIRED)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
-        FFMPEGCommon.cpp
-        VideoDecoder.h
-        VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
-    )
-    if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
-    else()
-        list(APPEND sources CPUOnlyDevice.cpp)
-    endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
+function(make_torchcodec_sublibrary
+    library_name
+    sources
+    dependent_libraries
+    ffmpeg_include_dirs)
 
-    target_include_directories(
-        ${library_name}
+    add_library(${library_name} SHARED ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
         PRIVATE
         ./../../../../
         "${TORCH_INSTALL_PREFIX}/include"
         ${Python3_INCLUDE_DIRS}
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: It's probably true that `libtorchcodec_decoder?.so` doesn't need the Python headers, but `libtorchcodec_pybind_ops?.so` does need the torch headers because it includes `VideoDecoder.h` which requires the torch headers.
created_at: 2025-03-20T00:53:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,62 +3,133 @@ project(TorchCodec)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+find_package(pybind11 REQUIRED)
 find_package(Torch REQUIRED)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
+
+function(make_torchcodec_sublibrary
+    library_name
+    type
+    sources
+    library_dependencies)
+
+    add_library(${library_name} ${type} ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
+        PRIVATE
+        ./../../../../
+        "${TORCH_INSTALL_PREFIX}/include"
+        ${Python3_INCLUDE_DIRS}
+    )
+
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
+    target_link_libraries(
+        ${library_name}
+        PUBLIC
+        ${library_dependencies}
+    )
+endfunction()
+
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target)
+
+    # TODO: List each library and its purpose.
+
+    # Create libtorchcodec_decoderN.so
+    set(decoder_library_name "libtorchcodec_decoder${ffmpeg_major_version}")
+    set(decoder_sources
+        AVIOContextHolder.cpp
         FFMPEGCommon.cpp
-        VideoDecoder.h
         VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
     )
+
     if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
+        list(APPEND decoder_sources CudaDevice.cpp)
     else()
-        list(APPEND sources CPUOnlyDevice.cpp)
+        list(APPEND decoder_sources CPUOnlyDevice.cpp)
     endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
 
-    target_include_directories(
-        ${library_name}
-        PRIVATE
-        ./../../../../
-        "${TORCH_INSTALL_PREFIX}/include"
-        ${Python3_INCLUDE_DIRS}
+    set(decoder_library_dependencies
+        ${ffmpeg_target}
+        ${TORCH_LIBRARIES}
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: Note that I took out `${Python3_LIBRARIES}` as a dependency on the decoder library. That caused problems with pybind11 importing on Mac: https://github.com/pybind/pybind11/issues/3907#issuecomment-1170412764. It's now only on the custom_ops library.
created_at: 2025-03-20T18:07:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,19 +31,37 @@ def load_torchcodec_extension():
     # correct .so file, so this for-loop succeeds on the first iteration.
 
     exceptions = []
+    pybind_ops_module_name = "_torchcodec_pybind_ops"
     for ffmpeg_major_version in (7, 6, 5, 4):
-        library_name = f"libtorchcodec{ffmpeg_major_version}"
+        decoder_library_name = f"libtorchcodec_decoder{ffmpeg_major_version}"
+        custom_ops_library_name = f"libtorchcodec_custom_ops{ffmpeg_major_version}"
+        pybind_ops_library_name = f"libtorchcodec_pybind_ops{ffmpeg_major_version}"
         try:
-            torch.ops.load_library(_get_extension_path(library_name))
+            torch.ops.load_library(_get_extension_path(decoder_library_name))
+            torch.ops.load_library(_get_extension_path(custom_ops_library_name))
+
+            pybind_ops_library_path = _get_extension_path(pybind_ops_library_name)
+            torch.ops.load_library(pybind_ops_library_path)
+            spec = importlib.util.spec_from_file_location(
+                pybind_ops_module_name,
+                pybind_ops_library_path,
+            )
+            if spec is None:
+                raise ImportError(
+                    f"Unable to load spec for module {pybind_ops_module_name} from path {pybind_ops_library_path}"
+                )
+
+            global _pybind_ops
+            _pybind_ops = importlib.util.module_from_spec(spec)
path: src/torchcodec/decoders/_core/ops.py
user: login: scotts
body: Works now! I'll make comments in the appropriate places.
created_at: 2025-03-20T18:08:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: NicolasHug
body: Shouldn't we be able to get both a pointer and the size from `py::bytes`? It might be simpler to keep `string`, but a small comment may help conveying the intent/reason.
created_at: 2025-03-21T11:15:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,62 +3,133 @@ project(TorchCodec)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+find_package(pybind11 REQUIRED)
 find_package(Torch REQUIRED)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
+
+function(make_torchcodec_sublibrary
+    library_name
+    type
+    sources
+    library_dependencies)
+
+    add_library(${library_name} ${type} ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
+        PRIVATE
+        ./../../../../
+        "${TORCH_INSTALL_PREFIX}/include"
+        ${Python3_INCLUDE_DIRS}
+    )
+
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
+    target_link_libraries(
+        ${library_name}
+        PUBLIC
+        ${library_dependencies}
+    )
+endfunction()
+
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target)
+
+    # TODO: List each library and its purpose.
+
+    # Create libtorchcodec_decoderN.so
+    set(decoder_library_name "libtorchcodec_decoder${ffmpeg_major_version}")
+    set(decoder_sources
+        AVIOContextHolder.cpp
         FFMPEGCommon.cpp
-        VideoDecoder.h
         VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
     )
+
     if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
+        list(APPEND decoder_sources CudaDevice.cpp)
     else()
-        list(APPEND sources CPUOnlyDevice.cpp)
+        list(APPEND decoder_sources CPUOnlyDevice.cpp)
     endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
 
-    target_include_directories(
-        ${library_name}
-        PRIVATE
-        ./../../../../
-        "${TORCH_INSTALL_PREFIX}/include"
-        ${Python3_INCLUDE_DIRS}
+    set(decoder_library_dependencies
+        ${ffmpeg_target}
+        ${TORCH_LIBRARIES}
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: IIUC the problem wasn't really with `libtorchcodec_decoder` (which shouldn't need Python anyway), it was with the `libtorchcodec_pybind_ops` library - which was pulling `libtorchcodec_decoder`?
created_at: 2025-03-21T11:50:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,62 +3,157 @@ project(TorchCodec)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+set(PYBIND11_FINDPYTHON ON)
+find_package(pybind11 REQUIRED)
 find_package(Torch REQUIRED)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
+
+function(make_torchcodec_sublibrary
+    library_name
+    type
+    sources
+    library_dependencies)
+
+    add_library(${library_name} ${type} ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
+        PRIVATE
+        ./../../../../
+        "${TORCH_INSTALL_PREFIX}/include"
+        ${Python3_INCLUDE_DIRS}
+    )
+
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
+    target_link_libraries(
+        ${library_name}
+        PUBLIC
+        ${library_dependencies}
+    )
+endfunction()
+
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target)
+
+    # We create three shared libraries per version of FFmpeg, where the version
+    # is denoted by N:
+    #
+    # 1. libtorchcodec_decoderN.{ext}: Base library which contains the
+    #    implementation of VideoDecoder and everything VideoDecoder needs. On
+    #    Linux, {ext} is so. On Mac, it is dylib.
+    #
+    # 2. libtorchcodec_custom_opsN.{ext}: Implementation of the PyTorch custom
+    #    ops. Depends on libtorchcodec_decoderN.{ext}. On Linux, {ext} is so.
+    #    On Mac, it is dylib.
+    #
+    # 3. libtorchcodec_pybind_opsN.{ext}: Implementation of the pybind11 ops. We
+    #    keep these separate from the PyTorch custom ops because we have to
+    #    load these libraries separately on the Python side. Depends on
+    #    libtorchcodec_decoderN.{ext}. On BOTH Linux and Mac {ext} is so.
+
+    # 1. Create libtorchcodec_decoderN.{ext}.
+    set(decoder_library_name "libtorchcodec_decoder${ffmpeg_major_version}")
+    set(decoder_sources
+        AVIOContextHolder.cpp
         FFMPEGCommon.cpp
-        VideoDecoder.h
         VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
     )
+
     if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
+        list(APPEND decoder_sources CudaDevice.cpp)
     else()
-        list(APPEND sources CPUOnlyDevice.cpp)
+        list(APPEND decoder_sources CPUOnlyDevice.cpp)
     endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
 
-    target_include_directories(
-        ${library_name}
-        PRIVATE
-        ./../../../../
-        "${TORCH_INSTALL_PREFIX}/include"
-        ${Python3_INCLUDE_DIRS}
+    set(decoder_library_dependencies
+        ${ffmpeg_target}
+        ${TORCH_LIBRARIES}
     )
 
-    set(NEEDED_LIBRARIES ${ffmpeg_target} ${TORCH_LIBRARIES}
-        ${Python3_LIBRARIES})
     if(ENABLE_CUDA)
-        list(APPEND NEEDED_LIBRARIES
-            ${CUDA_nppi_LIBRARY} ${CUDA_nppicc_LIBRARY} )
+        list(APPEND decoder_library_dependencies
+            ${CUDA_nppi_LIBRARY}
+            ${CUDA_nppicc_LIBRARY}
+        )
     endif()
-    target_link_libraries(
-        ${library_name}
+
+    make_torchcodec_sublibrary(
+        "${decoder_library_name}"
+        SHARED
+        "${decoder_sources}"
+        "${decoder_library_dependencies}"
+    )
+
+    # 2. Create libtorchcodec_custom_opsN.{ext}.
+    set(custom_ops_library_name "libtorchcodec_custom_ops${ffmpeg_major_version}")
+    set(custom_ops_sources
+        AVIOBytesContext.cpp
+        VideoDecoderOps.cpp
+    )
+    set(custom_ops_dependencies
+        ${decoder_library_name}
+        ${Python3_LIBRARIES}
+    )
+    make_torchcodec_sublibrary(
+        "${custom_ops_library_name}"
+        SHARED
+        "${custom_ops_sources}"
+        "${custom_ops_dependencies}"
+    )
+
+    # 3. Create libtorchcodec_pybind_opsN.so.
+    set(pybind_ops_library_name "libtorchcodec_pybind_ops${ffmpeg_major_version}")
+    set(pybind_ops_sources
+        AVIOFileLikeContext.cpp
+        PyBindOps.cpp
+    )
+    set(pybind_ops_dependencies
+       ${decoder_library_name}
+       pybind11::module
+    )
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: NicolasHug
body: If my understanding is correct, maybe add a comment like:

> In theory we should also add Python3_LIBRARIES as a dependency, but this is causing issues on MacOS. See `undefined dynamic_lookup` below.
created_at: 2025-03-21T11:52:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,68 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOFileLikeContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOFileLikeContext::AVIOFileLikeContext(py::object fileLike)
+    : fileLike_{UniquePyObject(new py::object(fileLike))} {
+  {
+    // TODO: Is it necessary to acquire the GIL here? Is it maybe even
+    // harmful? At the moment, this is only called from within a pybind
+    // function, and pybind guarantees we have the GIL.
+    py::gil_scoped_acquire gil;
+    TORCH_CHECK(
+        py::hasattr(fileLike, "read"),
+        "File like object must implement a read method.");
+    TORCH_CHECK(
+        py::hasattr(fileLike, "seek"),
+        "File like object must implement a seek method.");
+  }
+  createAVIOContext(&read, &seek, &fileLike_);
+}
+
+int AVIOFileLikeContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto fileLike = static_cast<UniquePyObject*>(opaque);
+
+  int num_read = 0;
+  while (num_read < buf_size) {
+    int request = buf_size - num_read;
+    // TODO: It is maybe more efficient to grab the lock once in the
+    // surrounding scope?
+    py::gil_scoped_acquire gil;
+    auto chunk = static_cast<std::string>(
+        static_cast<py::bytes>((*fileLike)->attr("read")(request)));
path: src/torchcodec/decoders/_core/AVIOFileLikeContext.cpp
user: login: scotts
body: We actually can't get the pointer. We can get the size from all `py::object` instances, but there's nothing to access the underlying data pointer. (Other than going through the actual `PyObject` struct, but I'd rather stick to the pybind11 API.)

However! Other people have asked the same thing: https://github.com/pybind/pybind11/issues/2517, which lead to someone else implementing something just as good, which is getting a `std::string_view`: https://github.com/pybind/pybind11/issues/1807. That will avoid the copy and get what we need. Good catch!
created_at: 2025-03-21T18:50:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,62 +3,133 @@ project(TorchCodec)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+find_package(pybind11 REQUIRED)
 find_package(Torch REQUIRED)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
 find_package(Python3 ${PYTHON_VERSION} EXACT COMPONENTS Development)
 
-function(make_torchcodec_library library_name ffmpeg_target)
-    set(
-        sources
-        FFMPEGCommon.h
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -pedantic -Werror ${TORCH_CXX_FLAGS}")
+
+function(make_torchcodec_sublibrary
+    library_name
+    type
+    sources
+    library_dependencies)
+
+    add_library(${library_name} ${type} ${sources})
+    set_target_properties(${library_name} PROPERTIES CXX_STANDARD 17)
+    target_include_directories(${library_name}
+        PRIVATE
+        ./../../../../
+        "${TORCH_INSTALL_PREFIX}/include"
+        ${Python3_INCLUDE_DIRS}
+    )
+
+    # Avoid adding the "lib" prefix which we already add explicitly.
+    set_target_properties(${library_name} PROPERTIES PREFIX "")
+
+    target_link_libraries(
+        ${library_name}
+        PUBLIC
+        ${library_dependencies}
+    )
+endfunction()
+
+function(make_torchcodec_libraries
+    ffmpeg_major_version
+    ffmpeg_target)
+
+    # TODO: List each library and its purpose.
+
+    # Create libtorchcodec_decoderN.so
+    set(decoder_library_name "libtorchcodec_decoder${ffmpeg_major_version}")
+    set(decoder_sources
+        AVIOContextHolder.cpp
         FFMPEGCommon.cpp
-        VideoDecoder.h
         VideoDecoder.cpp
-        VideoDecoderOps.h
-        VideoDecoderOps.cpp
-        DeviceInterface.h
     )
+
     if(ENABLE_CUDA)
-        list(APPEND sources CudaDevice.cpp)
+        list(APPEND decoder_sources CudaDevice.cpp)
     else()
-        list(APPEND sources CPUOnlyDevice.cpp)
+        list(APPEND decoder_sources CPUOnlyDevice.cpp)
     endif()
-    add_library(${library_name} SHARED ${sources})
-    set_property(TARGET ${library_name} PROPERTY CXX_STANDARD 17)
 
-    target_include_directories(
-        ${library_name}
-        PRIVATE
-        ./../../../../
-        "${TORCH_INSTALL_PREFIX}/include"
-        ${Python3_INCLUDE_DIRS}
+    set(decoder_library_dependencies
+        ${ffmpeg_target}
+        ${TORCH_LIBRARIES}
path: src/torchcodec/decoders/_core/CMakeLists.txt
user: login: scotts
body: Yes, because `libtorchcodec_pybind_ops` depends on `libtorchcodec_decoder`, and `libtorchcodec_decoder` used to depend on `${Python3_LIBRARIES}`, we ran into problems at runtime.

Now we're good because `libtorchcodec_decoder` does not depend on `${Python3_LIBRARIES}`, `libtorchcodec_custom_ops` does, and `libtorchcodec_pybind_ops` does not (it gets what it needs through `pybind11::module`).
created_at: 2025-03-21T19:06:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,70 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+// All rights reserved.
+//
+// This source code is licensed under the BSD-style license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include "src/torchcodec/decoders/_core/AVIOBytesContext.h"
+#include <torch/types.h>
+
+namespace facebook::torchcodec {
+
+AVIOBytesContext::AVIOBytesContext(const void* data, int64_t dataSize)
+    : dataContext_{static_cast<const uint8_t*>(data), dataSize, 0} {
+  TORCH_CHECK(data != nullptr, "Video data buffer cannot be nullptr!");
+  TORCH_CHECK(dataSize > 0, "Video data size must be positive");
+  createAVIOContext(&read, &seek, &dataContext_);
+}
+
+// The signature of this function is defined by FFMPEG.
+int AVIOBytesContext::read(void* opaque, uint8_t* buf, int buf_size) {
+  auto dataContext = static_cast<DataContext*>(opaque);
+  TORCH_CHECK(
+      dataContext->current <= dataContext->size,
+      "Tried to read outside of the buffer: current=",
+      dataContext->current,
+      ", size=",
+      dataContext->size);
+
+  int64_t numBytesRead = std::min(
path: src/torchcodec/decoders/_core/AVIOBytesContext.cpp
user: login: scotts
body: Note that this avoids the negative buffer problem. Before, we were doing a narrowing conversion: we casted the `int64_t` value to an `int` in order to compare it to an `int`. That caused a large positive value to become a negative value. It's safer to cast the `int` to an `int64_t` and then compare both values as `int64_t`. This is fine because `std::memcpy()` takes a `size_t`, which the `int64_t` can safely convert to.
created_at: 2025-03-21T20:01:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,30 +18,61 @@
     _get_extension_path,
 )
 
+_pybind_ops: Optional[ModuleType] = None
 
-def load_torchcodec_extension():
-    # Successively try to load libtorchcodec7.so, libtorchcodec6.so,
-    # libtorchcodec5.so, and libtorchcodec4.so. Each of these correspond to an
+
+def load_torchcodec_shared_libraries():
+    # Successively try to load libtorchcodec_*7.so, libtorchcodec_*6.so,
+    # libtorchcodec_*5.so, and libtorchcodec_*4.so. Each of these correspond to an
     # ffmpeg major version. This should cover all potential ffmpeg versions
     # installed on the user's machine.
     #
     # On fbcode, _get_extension_path() is overridden and directly points to the
     # correct .so file, so this for-loop succeeds on the first iteration.
+    #
+    # Note that we use two different methods for loading shared libraries:
+    #
+    #   1. torch.ops.load_library(): For PyTorch custom ops. Loading libraries
+    #      through PyTorch registers the custom ops with PyTorch's runtime and
+    #      the ops can be accessed through torch.ops after loading.
+    #
+    #   2. importlib: For pybind11 modules. We load them dynamically, rather
+    #      than using a plain import statement. A plain import statement only
+    #      works when the module name and file name match exactly, and the
+    #      shared library file is in the import path. Our shared libraries do
+    #      not meet those conditions.
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: Nit: I think we should remove this part from the sentence:

> and the shared library file is in the import path

Because the so/dylib files we produce are definitely in the "import path" (they're right next to other files we happily import). My point being that the above is *not* one of the conditions we fail to meet.
created_at: 2025-03-24T12:06:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,30 +18,61 @@
     _get_extension_path,
 )
 
+_pybind_ops: Optional[ModuleType] = None
 
-def load_torchcodec_extension():
-    # Successively try to load libtorchcodec7.so, libtorchcodec6.so,
-    # libtorchcodec5.so, and libtorchcodec4.so. Each of these correspond to an
+
+def load_torchcodec_shared_libraries():
+    # Successively try to load libtorchcodec_*7.so, libtorchcodec_*6.so,
+    # libtorchcodec_*5.so, and libtorchcodec_*4.so. Each of these correspond to an
     # ffmpeg major version. This should cover all potential ffmpeg versions
     # installed on the user's machine.
     #
     # On fbcode, _get_extension_path() is overridden and directly points to the
     # correct .so file, so this for-loop succeeds on the first iteration.
+    #
+    # Note that we use two different methods for loading shared libraries:
+    #
+    #   1. torch.ops.load_library(): For PyTorch custom ops. Loading libraries
path: src/torchcodec/decoders/_core/ops.py
user: login: NicolasHug
body: Nit: not just custom ops, it's also for the "pure C++" decoder library
created_at: 2025-03-24T12:07:19Z
author_association: CONTRIBUTOR

PR comments:
user: login: facebook-github-bot
created_at: 2025-03-24T14:12:19Z
body: @scotts has imported this pull request. If you are a Meta employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D71735990).
author_association: CONTRIBUTOR
user: login: facebook-github-bot
created_at: 2025-03-26T18:37:44Z
body: @scotts has imported this pull request. If you are a Meta employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D71735990).
author_association: CONTRIBUTOR