Repository: sktime/sktime

Issue:
[BUG] Tensorflow failing on macOS
When following development environment setup guide in sktime documentation, conda installation was not able to run properly on macOS doe to missing tensorflow-macos dependency.

To reproduce the issue run on macOS:
```shell
$ conda create -n sktime-dev python=3.8
$ conda activate sktime-dev
$ pip install -e ."[all_extras,dev]"
$ make test
zsh: illegal hardware instruction
```

A factor that can complicate things is that tensorflow-macos cannot be installed using Python 3.8.

Possible solutions can be adding a warning to sktime documentation or adjusting pyproject.toml to tackle this issue.

The expected behavior would be to run successfully all tests after initial installation.



Discussion:
Thanks for bringing this to our attention!

Quick question, since I don't fully get it:
* is this specific to a python version, or do you get it on all python versions?
* why does this not install the dependency automatically?
I looked a little bit more into the issue, and it may not be as easy to solve as I initially expected:

- Even if normal conda/miniconda can be installed on Mac ARM, they are not suited for them. Instead, the only alternative right now is [Miniforge](https://github.com/conda-forge/miniforge), which has a non-stable version prepared for ARM processors.
- tensorflow should not be installed. Instead, tensorflow-macos and tensorflow-metal should be the ones installed on Mac ARM.
- There are still some libraries that cannot be installed in Mac ARM, such as esig or tslearn.

Sadly, it seems that it will not be possible to use all_extras on Mac ARM until all these issues are solved.

The only good news is that this issues only present themselves if you use all_extras.

Also, the issues do not seem related to specific Python versions.

I do not know if it would be possible to add an all_extras_macos option to installation that avoids conflicting packages, or that would add too much clutter and more difficulty to maintain dependencies.

The conflicting libraries are:

- esig
- tensorflow (replaced by tensorflow-macos)
- tsfresh
- tslearn
- prophet

This would be an example of an alternative all-extras:
```requirements
all_extras_macos = [
    "tensorflow-macos",
    "tensorflow-metal",
    "cloudpickle",
    "dask",
    "dtw-python",
    "filterpy>=1.4.5",
    "h5py",
    "hmmlearn>=0.2.7",
    "gluonts>=0.9.0",
    "keras-self-attention",
    "matplotlib>=3.3.2",
    "mne",
    "pmdarima>=1.8.0,!=1.8.1,<3.0.0",
    "pykalman>=0.9.5",
    "pyod>=0.8.0",
    "scikit_posthocs>=0.6.5",
    "seaborn>=0.11.0",
    "statsforecast>=0.5.2",
    "stumpy>=1.5.1",
    "tbats>=1.1.0",
    "tensorflow-probability",
    "xarray"
]
```

Also, documentation should be adapted to explain this case or at least warn that Mac ARM hardware is not supported yet.
@danielli98, thanks for your investigation!

Regarding "how to", I think one can specify dependencies in `pyproject.toml` conditional on operating system, i.e., `tensorflow_macos; sys_platform == 'macos'` or similar. The precise syntax may be different but it could be found by search or experimentation.

For the "what", I think we need to fully pinpoint the failure condition here and verify it independently. Could you phrase your learnings as a condition in terms of:

* library affected
* package manager, e.g., `pip` or `conda` etc
* operating system, e.g., Windows, Mac and Unix
* processor or operating system, e.g., ARM vs Intel
* python version
* anything else that I may not have thought of

That would be hugely helpful!

If you don't know the precise condition (e.g., "I don't know whether it's the same on unix"), you can frame it as an "if" condition; of course, "only if" conditions are better but harder to learn about.

After reading your comment on how to fix the issue, it seems that using sys_platform specification on the affected requirements would be the best solution. Hopefully there will be documnetation on all the available options for that constraint.

Regarding the concrete scope of the issue:

- libraries affected:
    - esig: Cannot be installed
    - tensorflow Needs to be replaced by tensorflow-macos and tensorflow-metal packages
    - tsfresh: Cannot be installed
    - tslearn: Cannot be installed
    - prophet: Cannot be installed
- package manager: conda and pip (venv)
- operating system: macOS
- processor: ARM
- Python version: 3.8, 3.9, 3.10 (all supported versions by sktime)

The errors do not happen in Linux (Debian) x86 and Windows x86. I cannot say if there are similar issues in Linux ARM and Windows ARM, as I do not have any equipment with those configurations.
@dainelli98, thanks! Is this, per your understanding, "if" or "if and only if"?

That is, is this a combination of conditions where you know the issue occurs? Or do you *also* know that the issue does not occur under different conditions, e.g., macos but a different processor?
I tried with other setups (Linux, Windows) and this concrete issue happened only with the conditions described above.

The only setup I was not able to test was macOS with x86 processor (Intel), because of lack of that specific equipment.

I also cannot attest if the issue happens in Windows o Linux setups with ARM processors, but those setups are extremely uncommon as of now.
the one open question is: even if we change `pyproject` to this, how do we *test* the setup? We would like to make "install works" part of the CI, but I'm not sure, for instance, how to get a VM with an ARM processor, say.

Do you know that, @dainelli98?
To be honest, I do not know how to do that as of now. I have only worked on software deployed on x86 processors. It is only recently that I acquired an ARM powered device for personal projects

Certainly that would present an important issue, as I would complicate a lot thinks when it comes to automate the maintenance of the necessary adjustments in dependències that fixing this issue would required.

On the other hand I is likely that the number of developers using ARM powered devices will increase significantly in the next years, as Apple seems decided to keep using ARM chips for mac's.

In any case, even if it is decided that the issue cannot be fixed and maintained as of now, I would at least add a warning in documentation, so that new developers are aware.
hm - a note for developers is already here, in this issue - it is tagged "maintenance" and "bug".

For users of `all_extras`, where should the notification be?

As it currently stands, I infer:

* `all_extras` does not work on ARM processors
* it is currently not tested (since all CI tests currently pass)
I fully agree with the to things you point out to describe the issue.

Regarding where to put this information I am not sure where it would be the best place, as It has not been that long that I have been looking into this project. Maybe in the installation guide as a warning or a footnote.
yes, install guide, troubleshooting section might be a good place, or a footnote box.

Do you want to give it a try? Otherwise we can put it on the backlog with the millions of bugs...
`docs/source/installation.rst`
Yes, I'd be glad to add this information to the documentation myself.

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -42,6 +42,7 @@
 extensions = [
     "sphinx.ext.autodoc",
     "sphinx.ext.autosummary",
+    "sphinx.ext.autosectionlabel",
     "numpydoc",
     "sphinx.ext.intersphinx",
     "sphinx.ext.linkcode",  # link to GitHub source code via linkcode_resolve()


Test patch: