| Upgrade guide | |
| ############# | |
| This is a companion guide to the :doc:`changelog`. While the changelog briefly | |
| lists all of the new features, improvements and bug fixes, this upgrade guide | |
| focuses only the subset which directly impacts your experience when upgrading | |
| to a new version. But it goes into more detail. This includes things like | |
| deprecated APIs and their replacements, build system changes, general code | |
| modernization and other useful information. | |
| .. _upgrade-guide-2.11: | |
| v2.11 | |
| ===== | |
| * The minimum version of CMake is now 3.5. A future version will likely move to | |
| requiring something like CMake 3.15. Note that CMake 3.27 is removing the | |
| long-deprecated support for ``FindPythonInterp`` if you set 3.27 as the | |
| minimum or maximum supported version. To prepare for that future, CMake 3.15+ | |
| using ``FindPython`` or setting ``PYBIND11_FINDPYTHON`` is highly recommended, | |
| otherwise pybind11 will automatically switch to using ``FindPython`` if | |
| ``FindPythonInterp`` is not available. | |
| .. _upgrade-guide-2.9: | |
| v2.9 | |
| ==== | |
| * Any usage of the recently added ``py::make_simple_namespace`` should be | |
| converted to using ``py::module_::import("types").attr("SimpleNamespace")`` | |
| instead. | |
| * The use of ``_`` in custom type casters can now be replaced with the more | |
| readable ``const_name`` instead. The old ``_`` shortcut has been retained | |
| unless it is being used as a macro (like for gettext). | |
| .. _upgrade-guide-2.7: | |
| v2.7 | |
| ==== | |
| *Before* v2.7, ``py::str`` can hold ``PyUnicodeObject`` or ``PyBytesObject``, | |
| and ``py::isinstance<str>()`` is ``true`` for both ``py::str`` and | |
| ``py::bytes``. Starting with v2.7, ``py::str`` exclusively holds | |
| ``PyUnicodeObject`` (`#2409 <https://github.com/pybind/pybind11/pull/2409>`_), | |
| and ``py::isinstance<str>()`` is ``true`` only for ``py::str``. To help in | |
| the transition of user code, the ``PYBIND11_STR_LEGACY_PERMISSIVE`` macro | |
| is provided as an escape hatch to go back to the legacy behavior. This macro | |
| will be removed in future releases. Two types of required fixes are expected | |
| to be common: | |
| * Accidental use of ``py::str`` instead of ``py::bytes``, masked by the legacy | |
| behavior. These are probably very easy to fix, by changing from | |
| ``py::str`` to ``py::bytes``. | |
| * Reliance on py::isinstance<str>(obj) being ``true`` for | |
| ``py::bytes``. This is likely to be easy to fix in most cases by adding | |
| ``|| py::isinstance<bytes>(obj)``, but a fix may be more involved, e.g. if | |
| ``py::isinstance<T>`` appears in a template. Such situations will require | |
| careful review and custom fixes. | |
| .. _upgrade-guide-2.6: | |
| v2.6 | |
| ==== | |
| Usage of the ``PYBIND11_OVERLOAD*`` macros and ``get_overload`` function should | |
| be replaced by ``PYBIND11_OVERRIDE*`` and ``get_override``. In the future, the | |
| old macros may be deprecated and removed. | |
| ``py::module`` has been renamed ``py::module_``, but a backward compatible | |
| typedef has been included. This change was to avoid a language change in C++20 | |
| that requires unqualified ``module`` not be placed at the start of a logical | |
| line. Qualified usage is unaffected and the typedef will remain unless the | |
| C++ language rules change again. | |
| The public constructors of ``py::module_`` have been deprecated. Use | |
| ``PYBIND11_MODULE`` or ``module_::create_extension_module`` instead. | |
| An error is now thrown when ``__init__`` is forgotten on subclasses. This was | |
| incorrect before, but was not checked. Add a call to ``__init__`` if it is | |
| missing. | |
| A ``py::type_error`` is now thrown when casting to a subclass (like | |
| ``py::bytes`` from ``py::object``) if the conversion is not valid. Make a valid | |
| conversion instead. | |
| The undocumented ``h.get_type()`` method has been deprecated and replaced by | |
| ``py::type::of(h)``. | |
| Enums now have a ``__str__`` method pre-defined; if you want to override it, | |
| the simplest fix is to add the new ``py::prepend()`` tag when defining | |
| ``"__str__"``. | |
| If ``__eq__`` defined but not ``__hash__``, ``__hash__`` is now set to | |
| ``None``, as in normal CPython. You should add ``__hash__`` if you intended the | |
| class to be hashable, possibly using the new ``py::hash`` shortcut. | |
| The constructors for ``py::array`` now always take signed integers for size, | |
| for consistency. This may lead to compiler warnings on some systems. Cast to | |
| ``py::ssize_t`` instead of ``std::size_t``. | |
| The ``tools/clang`` submodule and ``tools/mkdoc.py`` have been moved to a | |
| standalone package, `pybind11-mkdoc`_. If you were using those tools, please | |
| use them via a pip install from the new location. | |
| The ``pybind11`` package on PyPI no longer fills the wheel "headers" slot - if | |
| you were using the headers from this slot, they are available by requesting the | |
| ``global`` extra, that is, ``pip install "pybind11[global]"``. (Most users will | |
| be unaffected, as the ``pybind11/include`` location is reported by ``python -m | |
| pybind11 --includes`` and ``pybind11.get_include()`` is still correct and has | |
| not changed since 2.5). | |
| .. _pybind11-mkdoc: https://github.com/pybind/pybind11-mkdoc | |
| CMake support: | |
| -------------- | |
| The minimum required version of CMake is now 3.4. Several details of the CMake | |
| support have been deprecated; warnings will be shown if you need to change | |
| something. The changes are: | |
| * ``PYBIND11_CPP_STANDARD=<platform-flag>`` is deprecated, please use | |
| ``CMAKE_CXX_STANDARD=<number>`` instead, or any other valid CMake CXX or CUDA | |
| standard selection method, like ``target_compile_features``. | |
| * If you do not request a standard, pybind11 targets will compile with the | |
| compiler default, but not less than C++11, instead of forcing C++14 always. | |
| If you depend on the old behavior, please use ``set(CMAKE_CXX_STANDARD 14 CACHE STRING "")`` | |
| instead. | |
| * Direct ``pybind11::module`` usage should always be accompanied by at least | |
| ``set(CMAKE_CXX_VISIBILITY_PRESET hidden)`` or similar - it used to try to | |
| manually force this compiler flag (but not correctly on all compilers or with | |
| CUDA). | |
| * ``pybind11_add_module``'s ``SYSTEM`` argument is deprecated and does nothing; | |
| linking now behaves like other imported libraries consistently in both | |
| config and submodule mode, and behaves like a ``SYSTEM`` library by | |
| default. | |
| * If ``PYTHON_EXECUTABLE`` is not set, virtual environments (``venv``, | |
| ``virtualenv``, and ``conda``) are prioritized over the standard search | |
| (similar to the new FindPython mode). | |
| In addition, the following changes may be of interest: | |
| * ``CMAKE_INTERPROCEDURAL_OPTIMIZATION`` will be respected by | |
| ``pybind11_add_module`` if set instead of linking to ``pybind11::lto`` or | |
| ``pybind11::thin_lto``. | |
| * Using ``find_package(Python COMPONENTS Interpreter Development)`` before | |
| pybind11 will cause pybind11 to use the new Python mechanisms instead of its | |
| own custom search, based on a patched version of classic ``FindPythonInterp`` | |
| / ``FindPythonLibs``. In the future, this may become the default. A recent | |
| (3.15+ or 3.18.2+) version of CMake is recommended. | |
| v2.5 | |
| ==== | |
| The Python package now includes the headers as data in the package itself, as | |
| well as in the "headers" wheel slot. ``pybind11 --includes`` and | |
| ``pybind11.get_include()`` report the new location, which is always correct | |
| regardless of how pybind11 was installed, making the old ``user=`` argument | |
| meaningless. If you are not using the function to get the location already, you | |
| are encouraged to switch to the package location. | |
| v2.2 | |
| ==== | |
| Deprecation of the ``PYBIND11_PLUGIN`` macro | |
| -------------------------------------------- | |
| ``PYBIND11_MODULE`` is now the preferred way to create module entry points. | |
| The old macro emits a compile-time deprecation warning. | |
| .. code-block:: cpp | |
| // old | |
| PYBIND11_PLUGIN(example) { | |
| py::module m("example", "documentation string"); | |
| m.def("add", [](int a, int b) { return a + b; }); | |
| return m.ptr(); | |
| } | |
| // new | |
| PYBIND11_MODULE(example, m) { | |
| m.doc() = "documentation string"; // optional | |
| m.def("add", [](int a, int b) { return a + b; }); | |
| } | |
| New API for defining custom constructors and pickling functions | |
| --------------------------------------------------------------- | |
| The old placement-new custom constructors have been deprecated. The new approach | |
| uses ``py::init()`` and factory functions to greatly improve type safety. | |
| Placement-new can be called accidentally with an incompatible type (without any | |
| compiler errors or warnings), or it can initialize the same object multiple times | |
| if not careful with the Python-side ``__init__`` calls. The new-style custom | |
| constructors prevent such mistakes. See :ref:`custom_constructors` for details. | |
| .. code-block:: cpp | |
| // old -- deprecated (runtime warning shown only in debug mode) | |
| py::class<Foo>(m, "Foo") | |
| .def("__init__", [](Foo &self, ...) { | |
| new (&self) Foo(...); // uses placement-new | |
| }); | |
| // new | |
| py::class<Foo>(m, "Foo") | |
| .def(py::init([](...) { // Note: no `self` argument | |
| return new Foo(...); // return by raw pointer | |
| // or: return std::make_unique<Foo>(...); // return by holder | |
| // or: return Foo(...); // return by value (move constructor) | |
| })); | |
| Mirroring the custom constructor changes, ``py::pickle()`` is now the preferred | |
| way to get and set object state. See :ref:`pickling` for details. | |
| .. code-block:: cpp | |
| // old -- deprecated (runtime warning shown only in debug mode) | |
| py::class<Foo>(m, "Foo") | |
| ... | |
| .def("__getstate__", [](const Foo &self) { | |
| return py::make_tuple(self.value1(), self.value2(), ...); | |
| }) | |
| .def("__setstate__", [](Foo &self, py::tuple t) { | |
| new (&self) Foo(t[0].cast<std::string>(), ...); | |
| }); | |
| // new | |
| py::class<Foo>(m, "Foo") | |
| ... | |
| .def(py::pickle( | |
| [](const Foo &self) { // __getstate__ | |
| return py::make_tuple(self.value1(), self.value2(), ...); // unchanged | |
| }, | |
| [](py::tuple t) { // __setstate__, note: no `self` argument | |
| return new Foo(t[0].cast<std::string>(), ...); | |
| // or: return std::make_unique<Foo>(...); // return by holder | |
| // or: return Foo(...); // return by value (move constructor) | |
| } | |
| )); | |
| For both the constructors and pickling, warnings are shown at module | |
| initialization time (on import, not when the functions are called). | |
| They're only visible when compiled in debug mode. Sample warning: | |
| .. code-block:: none | |
| pybind11-bound class 'mymodule.Foo' is using an old-style placement-new '__init__' | |
| which has been deprecated. See the upgrade guide in pybind11's docs. | |
| Stricter enforcement of hidden symbol visibility for pybind11 modules | |
| --------------------------------------------------------------------- | |
| pybind11 now tries to actively enforce hidden symbol visibility for modules. | |
| If you're using either one of pybind11's :doc:`CMake or Python build systems | |
| <compiling>` (the two example repositories) and you haven't been exporting any | |
| symbols, there's nothing to be concerned about. All the changes have been done | |
| transparently in the background. If you were building manually or relied on | |
| specific default visibility, read on. | |
| Setting default symbol visibility to *hidden* has always been recommended for | |
| pybind11 (see :ref:`faq:symhidden`). On Linux and macOS, hidden symbol | |
| visibility (in conjunction with the ``strip`` utility) yields much smaller | |
| module binaries. `CPython's extension docs`_ also recommend hiding symbols | |
| by default, with the goal of avoiding symbol name clashes between modules. | |
| Starting with v2.2, pybind11 enforces this more strictly: (1) by declaring | |
| all symbols inside the ``pybind11`` namespace as hidden and (2) by including | |
| the ``-fvisibility=hidden`` flag on Linux and macOS (only for extension | |
| modules, not for embedding the interpreter). | |
| .. _CPython's extension docs: https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module | |
| The namespace-scope hidden visibility is done automatically in pybind11's | |
| headers and it's generally transparent to users. It ensures that: | |
| * Modules compiled with different pybind11 versions don't clash with each other. | |
| * Some new features, like ``py::module_local`` bindings, can work as intended. | |
| The ``-fvisibility=hidden`` flag applies the same visibility to user bindings | |
| outside of the ``pybind11`` namespace. It's now set automatic by pybind11's | |
| CMake and Python build systems, but this needs to be done manually by users | |
| of other build systems. Adding this flag: | |
| * Minimizes the chances of symbol conflicts between modules. E.g. if two | |
| unrelated modules were statically linked to different (ABI-incompatible) | |
| versions of the same third-party library, a symbol clash would be likely | |
| (and would end with unpredictable results). | |
| * Produces smaller binaries on Linux and macOS, as pointed out previously. | |
| Within pybind11's CMake build system, ``pybind11_add_module`` has always been | |
| setting the ``-fvisibility=hidden`` flag in release mode. From now on, it's | |
| being applied unconditionally, even in debug mode and it can no longer be opted | |
| out of with the ``NO_EXTRAS`` option. The ``pybind11::module`` target now also | |
| adds this flag to its interface. The ``pybind11::embed`` target is unchanged. | |
| The most significant change here is for the ``pybind11::module`` target. If you | |
| were previously relying on default visibility, i.e. if your Python module was | |
| doubling as a shared library with dependents, you'll need to either export | |
| symbols manually (recommended for cross-platform libraries) or factor out the | |
| shared library (and have the Python module link to it like the other | |
| dependents). As a temporary workaround, you can also restore default visibility | |
| using the CMake code below, but this is not recommended in the long run: | |
| .. code-block:: cmake | |
| target_link_libraries(mymodule PRIVATE pybind11::module) | |
| add_library(restore_default_visibility INTERFACE) | |
| target_compile_options(restore_default_visibility INTERFACE -fvisibility=default) | |
| target_link_libraries(mymodule PRIVATE restore_default_visibility) | |
| Local STL container bindings | |
| ---------------------------- | |
| Previous pybind11 versions could only bind types globally -- all pybind11 | |
| modules, even unrelated ones, would have access to the same exported types. | |
| However, this would also result in a conflict if two modules exported the | |
| same C++ type, which is especially problematic for very common types, e.g. | |
| ``std::vector<int>``. :ref:`module_local` were added to resolve this (see | |
| that section for a complete usage guide). | |
| ``py::class_`` still defaults to global bindings (because these types are | |
| usually unique across modules), however in order to avoid clashes of opaque | |
| types, ``py::bind_vector`` and ``py::bind_map`` will now bind STL containers | |
| as ``py::module_local`` if their elements are: builtins (``int``, ``float``, | |
| etc.), not bound using ``py::class_``, or bound as ``py::module_local``. For | |
| example, this change allows multiple modules to bind ``std::vector<int>`` | |
| without causing conflicts. See :ref:`stl_bind` for more details. | |
| When upgrading to this version, if you have multiple modules which depend on | |
| a single global binding of an STL container, note that all modules can still | |
| accept foreign ``py::module_local`` types in the direction of Python-to-C++. | |
| The locality only affects the C++-to-Python direction. If this is needed in | |
| multiple modules, you'll need to either: | |
| * Add a copy of the same STL binding to all of the modules which need it. | |
| * Restore the global status of that single binding by marking it | |
| ``py::module_local(false)``. | |
| The latter is an easy workaround, but in the long run it would be best to | |
| localize all common type bindings in order to avoid conflicts with | |
| third-party modules. | |
| Negative strides for Python buffer objects and numpy arrays | |
| ----------------------------------------------------------- | |
| Support for negative strides required changing the integer type from unsigned | |
| to signed in the interfaces of ``py::buffer_info`` and ``py::array``. If you | |
| have compiler warnings enabled, you may notice some new conversion warnings | |
| after upgrading. These can be resolved using ``static_cast``. | |
| Deprecation of some ``py::object`` APIs | |
| --------------------------------------- | |
| To compare ``py::object`` instances by pointer, you should now use | |
| ``obj1.is(obj2)`` which is equivalent to ``obj1 is obj2`` in Python. | |
| Previously, pybind11 used ``operator==`` for this (``obj1 == obj2``), but | |
| that could be confusing and is now deprecated (so that it can eventually | |
| be replaced with proper rich object comparison in a future release). | |
| For classes which inherit from ``py::object``, ``borrowed`` and ``stolen`` | |
| were previously available as protected constructor tags. Now the types | |
| should be used directly instead: ``borrowed_t{}`` and ``stolen_t{}`` | |
| (`#771 <https://github.com/pybind/pybind11/pull/771>`_). | |
| Stricter compile-time error checking | |
| ------------------------------------ | |
| Some error checks have been moved from run time to compile time. Notably, | |
| automatic conversion of ``std::shared_ptr<T>`` is not possible when ``T`` is | |
| not directly registered with ``py::class_<T>`` (e.g. ``std::shared_ptr<int>`` | |
| or ``std::shared_ptr<std::vector<T>>`` are not automatically convertible). | |
| Attempting to bind a function with such arguments now results in a compile-time | |
| error instead of waiting to fail at run time. | |
| ``py::init<...>()`` constructor definitions are also stricter and now prevent | |
| bindings which could cause unexpected behavior: | |
| .. code-block:: cpp | |
| struct Example { | |
| Example(int &); | |
| }; | |
| py::class_<Example>(m, "Example") | |
| .def(py::init<int &>()); // OK, exact match | |
| // .def(py::init<int>()); // compile-time error, mismatch | |
| A non-``const`` lvalue reference is not allowed to bind to an rvalue. However, | |
| note that a constructor taking ``const T &`` can still be registered using | |
| ``py::init<T>()`` because a ``const`` lvalue reference can bind to an rvalue. | |
| v2.1 | |
| ==== | |
| Minimum compiler versions are enforced at compile time | |
| ------------------------------------------------------ | |
| The minimums also apply to v2.0 but the check is now explicit and a compile-time | |
| error is raised if the compiler does not meet the requirements: | |
| * GCC >= 4.8 | |
| * clang >= 3.3 (appleclang >= 5.0) | |
| * MSVC >= 2015u3 | |
| * Intel C++ >= 15.0 | |
| The ``py::metaclass`` attribute is not required for static properties | |
| --------------------------------------------------------------------- | |
| Binding classes with static properties is now possible by default. The | |
| zero-parameter version of ``py::metaclass()`` is deprecated. However, a new | |
| one-parameter ``py::metaclass(python_type)`` version was added for rare | |
| cases when a custom metaclass is needed to override pybind11's default. | |
| .. code-block:: cpp | |
| // old -- emits a deprecation warning | |
| py::class_<Foo>(m, "Foo", py::metaclass()) | |
| .def_property_readonly_static("foo", ...); | |
| // new -- static properties work without the attribute | |
| py::class_<Foo>(m, "Foo") | |
| .def_property_readonly_static("foo", ...); | |
| // new -- advanced feature, override pybind11's default metaclass | |
| py::class_<Bar>(m, "Bar", py::metaclass(custom_python_type)) | |
| ... | |
| v2.0 | |
| ==== | |
| Breaking changes in ``py::class_`` | |
| ---------------------------------- | |
| These changes were necessary to make type definitions in pybind11 | |
| future-proof, to support PyPy via its ``cpyext`` mechanism (`#527 | |
| <https://github.com/pybind/pybind11/pull/527>`_), and to improve efficiency | |
| (`rev. 86d825 <https://github.com/pybind/pybind11/commit/86d825>`_). | |
| 1. Declarations of types that provide access via the buffer protocol must | |
| now include the ``py::buffer_protocol()`` annotation as an argument to | |
| the ``py::class_`` constructor. | |
| .. code-block:: cpp | |
| py::class_<Matrix>("Matrix", py::buffer_protocol()) | |
| .def(py::init<...>()) | |
| .def_buffer(...); | |
| 2. Classes which include static properties (e.g. ``def_readwrite_static()``) | |
| must now include the ``py::metaclass()`` attribute. Note: this requirement | |
| has since been removed in v2.1. If you're upgrading from 1.x, it's | |
| recommended to skip directly to v2.1 or newer. | |
| 3. This version of pybind11 uses a redesigned mechanism for instantiating | |
| trampoline classes that are used to override virtual methods from within | |
| Python. This led to the following user-visible syntax change: | |
| .. code-block:: cpp | |
| // old v1.x syntax | |
| py::class_<TrampolineClass>("MyClass") | |
| .alias<MyClass>() | |
| ... | |
| // new v2.x syntax | |
| py::class_<MyClass, TrampolineClass>("MyClass") | |
| ... | |
| Importantly, both the original and the trampoline class are now specified | |
| as arguments to the ``py::class_`` template, and the ``alias<..>()`` call | |
| is gone. The new scheme has zero overhead in cases when Python doesn't | |
| override any functions of the underlying C++ class. | |
| `rev. 86d825 <https://github.com/pybind/pybind11/commit/86d825>`_. | |
| The class type must be the first template argument given to ``py::class_`` | |
| while the trampoline can be mixed in arbitrary order with other arguments | |
| (see the following section). | |
| Deprecation of the ``py::base<T>()`` attribute | |
| ---------------------------------------------- | |
| ``py::base<T>()`` was deprecated in favor of specifying ``T`` as a template | |
| argument to ``py::class_``. This new syntax also supports multiple inheritance. | |
| Note that, while the type being exported must be the first argument in the | |
| ``py::class_<Class, ...>`` template, the order of the following types (bases, | |
| holder and/or trampoline) is not important. | |
| .. code-block:: cpp | |
| // old v1.x | |
| py::class_<Derived>("Derived", py::base<Base>()); | |
| // new v2.x | |
| py::class_<Derived, Base>("Derived"); | |
| // new -- multiple inheritance | |
| py::class_<Derived, Base1, Base2>("Derived"); | |
| // new -- apart from `Derived` the argument order can be arbitrary | |
| py::class_<Derived, Base1, Holder, Base2, Trampoline>("Derived"); | |
| Out-of-the-box support for ``std::shared_ptr`` | |
| ---------------------------------------------- | |
| The relevant type caster is now built in, so it's no longer necessary to | |
| include a declaration of the form: | |
| .. code-block:: cpp | |
| PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>) | |
| Continuing to do so won't cause an error or even a deprecation warning, | |
| but it's completely redundant. | |
| Deprecation of a few ``py::object`` APIs | |
| ---------------------------------------- | |
| All of the old-style calls emit deprecation warnings. | |
| +---------------------------------------+---------------------------------------------+ | |
| | Old syntax | New syntax | | |
| +=======================================+=============================================+ | |
| | ``obj.call(args...)`` | ``obj(args...)`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``obj.str()`` | ``py::str(obj)`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``auto l = py::list(obj); l.check()`` | ``py::isinstance<py::list>(obj)`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``py::object(ptr, true)`` | ``py::reinterpret_borrow<py::object>(ptr)`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``py::object(ptr, false)`` | ``py::reinterpret_steal<py::object>(ptr)`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``if (obj.attr("foo"))`` | ``if (py::hasattr(obj, "foo"))`` | | |
| +---------------------------------------+---------------------------------------------+ | |
| | ``if (obj["bar"])`` | ``if (obj.contains("bar"))`` | | |
| +---------------------------------------+---------------------------------------------+ | |