Repository: AppDaemon/appdaemon (language: python)

Issue:
Improve packaging system
The `requirements.txt` file is used in a bit confusing way. It it usual practice for this file to contain all the dependencies present in the project virtualenv, created using `pip freeze > requirements.txt`, so that the virtualenv can be easily re-created by any developer locally using `pip install -r requirements.txt.`
But this file is also used as the `install_requires` inside `setup.py`, creating a bit of a conflict.
If one installs the appdaemon locally with `pip install .`, then run `pip freeze > requirements.txt`, the `requirements.txt` file is modified, since all the transitive dependencies are also loaded, as is expected. So the question is: shouldn't the declaration of the project dependencies be moved in a more appropriate place, separate from `requirements.txt`?

## Proposed solution
I propose to more crealry declare the project dependencies, by upgrading the way we use the Python packaging system.
At the moment the package is created from the `setup.py`, read by `setuptools` when building the package.
The way to package applications has changed a lot over the years. Nowadays if one were to follow the official [Python packaging guide](https://packaging.python.org/en/latest/tutorials/packaging-projects/), the recommended setup is to use a `pyproject.toml` to describe all the metadata of the project, including the build tool and the project dependencies.
I suggest we move from the current `setup.py` to a more modern and "standard" solution, by migrating to a `pyproject.toml` file to contain all the project dependencies.



Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 465b3e7a9..2a89a403a 100644
--- a/README.md
+++ b/README.md
@@ -8,19 +8,18 @@ practically any event driven application.
 It also provides a configurable dashboard (HADashboard)
 suitable for wall mounted tablets.
 
-# Release Cycle Frequency
+## Getting started
+
+For full instructions on installation and usage check out the [AppDaemon Project Documentation](http://appdaemon.readthedocs.io).
+
+## Release Cycle Frequency
 
 AppDaemon has reached a very stable point, works reliably and is fairly feature rich at this point
 in its development. For that reason, releases have been slow in recent months. This does not mean that AppDaemon has been abandoned -
  it is used every day by the core developers and has an active discord server [here](https://discord.gg/qN7c7JcFjk) - please join us for tips
 and tricks, AppDaemon discussions and general home automation.
 
-For full instructions on installation and use check out the [AppDaemon Project Documentation](http://appdaemon.readthedocs.io).
-
-## Development of the AppDaemon library
+## Contributing
 
-We use [pre-commit](https://pre-commit.com) for linting of the code, so `pip install pre_commit` and run
-```
-pre-commit install
-```
-in the repository.
+This is an active open-source project. We are always open to people who want to use the code or contribute to it. Thank you for being involved!
+Have a look at the [official documentation](https://appdaemon.readthedocs.io/en/latest/DEV.html) for more information.
diff --git a/conf/.gitignore b/conf/.gitignore
index 77356c314..121548b67 100644
--- a/conf/.gitignore
+++ b/conf/.gitignore
@@ -1,1 +1,3 @@
 test
+# config file used during development
+appdaemon.yaml
\ No newline at end of file
diff --git a/conf/appdaemon.yaml.example b/conf/appdaemon.yaml.example
index 839cfdbc9..5f0fabdf8 100644
--- a/conf/appdaemon.yaml.example
+++ b/conf/appdaemon.yaml.example
@@ -10,7 +10,7 @@ appdaemon:
       token:
       cert_verify: True
 http:
-  url:
+  url: http://0.0.0.0:5050
 admin:
 api:
 hadashboard:
diff --git a/docs/DEV.rst b/docs/DEV.rst
index d3e107011..e93b4480c 100644
--- a/docs/DEV.rst
+++ b/docs/DEV.rst
@@ -1,104 +1,147 @@
 Development
 ===========
-
 If you want to help with the development of AppDaemon all assistance is gratefully received! Here are a few things you can do to help.
 
-Running a Dev Version
----------------------
+Installing a beta version
+-------------------------
+For the adventurous among you, it is possible to run a pre-release version to get a preview of changes before they are released as part of a stable build.
+**Please be aware**: use it at your own risk.  Although we try to keep things consistent and functional, we can't guarantee that things won't break.
+However, feedback from brave souls running this pre-release version is always gratefully received!
+
+Also, note, that to run a development version you should be using the *Pip install method*. Docker builds are created for dev too, but there is no hass.io support.
 
-For the adventurous among you, it is possible to run the very latest dev code to get a preview of changes before they are released as part of a stable build. Be aware you use the ``dev`` branch at your own risk.  Although we try to keep things consistent and functional, we can't guarantee that things won't break. However, feedback from brave souls running the dev branch is always gratefully received!
+There are 2 different ways of installing via Pip. If we are running a beta, we will have a number of specific milestone builds.
+The beta version will not install by default using the standard ``pip`` command but can be installed if its exact version is specified to `pip``:
 
-Also, note, that to run a dev version you should be using the *PIP install method*. Docker builds are created for dev too, but there is no hass.io support.
+.. code:: console
 
-There are 2 different ways of installing via PIP. If we are running a beta, we will have a number of specific milestone builds. These will not install by default using the standard pip command line but can be installed if their exact version is given to the pip command:
+    $ pip install appdaemon==<beta version>
 
-.. code:: bash
+Setting up a development environment
+------------------------------------
+If you want to run the latest code available in the ``dev`` branch, or if you want to run a local version of the application separate from your existing installation, take the following steps:
 
-    $ pip3 install appdaemon==<specific beta version>
+Clone the repository
+^^^^^^^^^^^^^^^^^^^^
+Download a copy of the ``dev`` branch. Run the following command to clone the ``dev`` branch of the official `AppDaemon repository <https://github.com/AppDaemon/appdaemon.git>`_:
 
-For non-beta builds, when the goal is just to run the latest dev code, or if you want a local version of the dev code without installing it over the top of the stable code, take the following steps:
+.. code:: console
 
-Clone the Repository
-~~~~~~~~~~~~~~~~~~~~
+    $ git clone -b dev https://github.com/AppDaemon/appdaemon.git
 
-First, we need to get a clean copy of the dev branch. To do this, create a new directory, and change into it. Run the following command to clone the dev branch of the AppDaemon repository:
+This will create a directory called ``appdaemon``: this is your local Git repository, and all subsequent commands will need to be run from inside it.
 
-.. code:: bash
+Requirements
+^^^^^^^^^^^^
+Firstly, it is recommended to create a Python virtual environment (VE) and enable it. The best practice here is to use a VE specifically for the development version.
+In some cases, it is possible that the ``dev`` branch may have updated dependencies that will be incompatible with the latest stable release, and may break it.
 
-    $ git clone -b dev https://github.com/AppDaemon/appdaemon.git
+Make sure you are in the ``appdaemon`` project directory, then run the following commands:
 
-This will create a directory called ``appdaemon`` - this is your repository directory, and all commands will need to be run from inside it.
+1. Install the project dependencies, along with the development dependencies
 
-Run AppDaemon from the command line
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code:: console
 
-Now that you have a local copy of the code, the next step is to run AppDaemon using that code.
+    $ pip install -e .[dev]
 
-As a first step, if you are using a Virtual Environment (VE) enable it. The best practice here is to use a VE specifically for the dev version. In some cases, it is possible that the dev branch may have updated dependencies that will be incompatible with the latest stable release, and may break it. In this process, you can find dependency issues, review ``setup.py`` for a list of required dependencies.
+1. Setup `pre-commit hooks <https://pre-commit.com>`_. This will make sure that modified files are linted and formatted before every commit.
 
-To run the cloned version of AppDaemon, make sure you are in the ``appdaemon`` subdirectory and run the following command:
+.. code:: console
 
-.. code:: bash
+    $ pre-commit install
 
-    $ python3 -m appdaemon -c <PATH To CONFIG DIRECTORY>
+Running the application
+^^^^^^^^^^^^^^^^^^^^^^^
+Now that you have a local copy of the source code, the next step is to run AppDaemon.
 
-In most cases, it is possible to share config directories with other AppDaemon instances. However,  you must be aware of apps that use new features as they will likely cause errors for the stable version. If you prefer, you can create an entirely new conf directory for your dev environment.
+Copy the default configuration file (edit it if you need to tweak some settings):
 
-Install AppDamon via PIP (Optional)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code:: console
 
-Although the recommended way of running a dev build is to use the command line above, it is possible to install an AppDaemon dev build as a pip package. If you do so, it will replace your stable version, so only do this if you are confident with packages and VEs. However,  if you use a specific VE for the dev build, this should not be an issue. Also, remember that if you do this, you will need to reinstall the package as an extra step every time you refresh the dev repository (see below).
+    $ cp conf/appdaemon.yaml.example conf/appdaemon.yaml
 
-To install the dev build as a package, change to the ``appdaemon`` directory and run the following command:
+Start the application:
 
-.. code:: bash
+.. code:: console
 
-    $ pip3 install .
+    $ python -m appdaemon -c conf/
 
-Updating AppDaemon to the latest dev version
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In most cases, it is possible to share configuration directories with other AppDaemon instances.
+However, you must be aware of AppDaemon apps that use new features as they will likely cause errors for the other pre-existing version.
+It is recommended to use an entirely separate configuration directory for your development environment.
 
-When the dev version has been updated, and you want to pull over the latest changes, run the following command from the ``appdaemon`` directory:
+Getting the latest changes
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+When there are updates on the ``dev`` branch and you want to pull over the latest changes, run the following command from the ``appdaemon`` directory:
 
-.. code:: bash
+.. code:: console
 
     $ git pull
 
-You can then immediately run the latest version with the command line above. If you are using pip, remember to rerun the ``install`` command using the ``--upgrade flag``:
+You can then immediately run the latest version with the commands previously detailed.
 
-.. code:: bash
+Building a distribution package
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+To build a Python distribution package (*wheel*), run the following command:
 
-    $ pip3 install --upgrade .
+.. code:: console
 
-Pull Requests
--------------
+    $ python -m build
+
+It will output the result of the build inside a ``dist/`` folder.
+
+The package can be installed directly via pip:
 
-If you see a way to improve on AppDaemon, We are pleased to receive Pull Requests. The official AppDaemon repository is here:
+.. code:: console
 
-https://github.com/AppDaemon/appdaemon
+    $ pip install dist/appdaemon*.whl
 
-Please note, if documentation is required to make sense of the PR, the PR will not be accepted without it.
+Project structure
+-----------------
 
-Documentation
+The Python project follows the conventional PEP 621, using a ``pyproject.toml`` to define its metadata.
+The repository is divided into various folder:
+
+appdaemon
+    source code of the Python package
+docs
+    source code from which this documentation is built
+tests
+    unit tests written with ``pytest``
+conf
+    configuration directory, containing some sample files
+
+
+Pull Requests
 -------------
 
-Assistance with the docs is always welcome, whether its fixing typos and incorrect information or reorganizing and
-adding to the docs to make them more helpful. To work on the docs, submit a pull request with the changes, and I
-will review and merge them in the usual way. I use `readthedocs` to build and host the docs, and you can easily
-set up a preview of your edits as follows:
+If you would like to improve AppDaemon, we are pleased to receive Pull Requests in `the official AppDaemon repository <https://github.com/AppDaemon/appdaemon>`_.
+
+Please note, if some documentation is required to make sense of the PR, the PR will not be accepted without it.
+
+Working on the documentation
+----------------------------
+
+Assistance with the docs is always welcome, whether its fixing typos and incorrect information or reorganizing and adding to the docs to make them more helpful.
+To work on the docs, submit a pull request with the changes, and I
+will review and merge them in the usual way.
+I use `Read the Docs <https://readthedocs.org/>`_ to build and host the documentation pages.
+You can easily preview your edits locally, by running the following command:
 
-First, install sphinx and support libraries.
+If not already done, install the development dependencies locally.
+The following command downloads and install the optional dependencies, as defined in the `pyproject.toml` file:
 
-.. code:: bash
+.. code:: console
 
-    $ pip3 install sphinx sphinx-rtd-theme sphinx-autobuild
+    $ pip install .[dev]
 
 Then `cd` to the `docs` subdirectory, where all the `rst` files are found, and run the following command:
 
-.. code:: bash
+.. code:: console
 
-    $ sphinx-autobuild  --host=0.0.0.0  . _build_html
+    $ sphinx-autobuild --host=0.0.0.0 docs/ docs/_build/html
 
-Sphinx will take a minute or so to build the current version of the docs, and it will then be available on port 8000
-(e.g., http://localhost:8080) of the machine hosting sphinx. As you make changes. Sphinx will automatically detect them
-and update the browser page in real-time. When you finish your editing, stop sphinx by typing ctrl-c.
+Sphinx will take a minute or so to build the current version of the docs, and it will then be available on local port 8000
+(e.g., http://localhost:8000).
+As you make changes, sphinx will automatically detects them and updates the browser page in real-time.
+When you finish your edit, you can stop the server via ``Ctrl-C``.
diff --git a/pyproject.toml b/pyproject.toml
index 26a9add67..3698f5642 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,5 +1,85 @@
 [project]
 name = "appdaemon"
+description="Apps for the Home Assistant home automation package."
+readme = "README.md"
+authors = [
+    {name = "Andrew I Cockburn", email = "appdaemon@acockburn.com"},
+]
+dynamic = ["version"] # The verison is computed dynamically. See `tool.setuptools.dynamic` section below.
+license= { file = "LICENSE.md" }
+requires-python = ">=3.7"
+keywords=[ "appdaemon", "home", "automation" ]
+dependencies = [
+    'importlib-metadata; python_version<"3.8"',
+    'aiohttp ~= 3.8.4',
+    'aiohttp_jinja2 ~= 1.5.1',
+    'astral ~= 3.2',
+    'bcrypt ~= 4.0.1',
+    'deepdiff ~= 6.2.3',
+    'feedparser ~= 6.0.10',
+    'iso8601 ~= 1.1.0',
+    'paho-mqtt ~= 1.6.1',
+    'pid ~= 3.0.4',
+    'python-dateutil ~= 2.8.2',
+    'python-socketio ~= 5.5.0',
+    'pytz ~= 2022.7.1',
+    'pyyaml ~= 6.0',
+    'requests ~= 2.28.2',
+    'sockjs ~= 0.11.0',
+    'uvloop==0.17.0; sys_platform != "win32"',
+    'websocket-client ~= 1.5.1'
+]
+
+classifiers= [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Natural Language :: English",
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Topic :: Home Automation",
+]
+
+[project.urls]
+documentation = "https://appdaemon.readthedocs.io"
+repository = "https://github.com/home-assistant/appdaemon.git"
+changelog = "https://github.com/AppDaemon/appdaemon/blob/master/docs/HISTORY.rst"
+
+
+# Declare development dependencies as optional
+[project.optional-dependencies]
+dev = [
+    "build==0.10.0",
+    "flake8==3.7.8",
+    "black==22.3.0",
+    'pre-commit==3.1.1; python_version>"3.7"',  # pre-commit does not support Python < 3.8
+    "pytest==7.2.1",
+    "sphinx-autobuild==2021.3.14",
+    "sphinx-rtd-theme==1.2.0",
+    "pygments==2.14.0"
+]
+
+
+# Define the main CLI script
+[project.scripts]
+appdaemon = "appdaemon.__main__:main"
+
+# Use setuptools as the build system
+[build-system]
+requires = ["setuptools >= 61.0.0"]
+build-backend = "setuptools.build_meta"
+
+# Tell setuptools how to find the package source files
+# https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
+[tool.setuptools.packages.find]
+exclude = ["contrib", "docs", "tests*"]
+
+# Dynamically read version from version.py module
+# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata
+[tool.setuptools.dynamic]
+version = {attr = "appdaemon.version.__version__"}
 
 # Taken from: https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html
 [tool.pytest.ini_options]
diff --git a/requirements.txt b/requirements.txt
deleted file mode 100644
index 14ec4dff1..000000000
--- a/requirements.txt
+++ /dev/null
@@ -1,24 +0,0 @@
-astral==3.2
-pytz==2022.7.1
-requests==2.28.2
-websocket-client==1.5.1
-aiohttp==3.8.4
-aiodns==3.0.0
-python-engineio==4.3.4
-python-socketio==5.5.0
-yarl==1.8.2
-Jinja2==3.1.2
-aiohttp_jinja2==1.5.1
-pyyaml==6.0
-voluptuous==0.13.1
-feedparser==6.0.10
-iso8601==1.1.0
-bcrypt==4.0.1
-paho-mqtt==1.6.1
-deepdiff==6.2.3
-python-dateutil==2.8.2
-pid==3.0.4
-sockjs==0.11.0
-pygments==2.14.0
-wheel==0.38.4
-uvloop==0.17.0; sys_platform != 'win32'
diff --git a/setup.py b/setup.py
deleted file mode 100755
index e83fc8644..000000000
--- a/setup.py
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-from setuptools import setup, find_packages
-
-from appdaemon.version import __version__
-
-# sudo apt-get install python3-aiohttp-dbg
-
-with open("requirements.txt") as f:
-    install_requires = [x for x in f.read().split("\n") if x]
-
-with open("README.md") as f:
-    long_description = f.read()
-
-setup(
-    name="appdaemon",
-    version=__version__,
-    description="Apps for the Home Assistant home automation package.",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    author="Andrew I Cockburn",
-    author_email="appdaemon@acockburn.com",
-    url="https://github.com/home-assistant/appdaemon.git",
-    packages=find_packages(exclude=["contrib", "docs", "tests*"]),
-    include_package_data=True,
-    install_requires=install_requires,
-    license="Apache License 2.0",
-    python_requires=">=3.7",
-    zip_safe=False,
-    keywords=["appdaemon", "home", "automation"],
-    entry_points={"console_scripts": ["appdaemon = appdaemon.__main__:main"]},
-    classifiers=[
-        "Development Status :: 5 - Production/Stable",
-        "Intended Audience :: Developers",
-        "License :: OSI Approved :: Apache Software License",
-        "Natural Language :: English",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Topic :: Home Automation",
-    ],
-)


Test patch:
diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml
index 478c79c7a..93c66c62d 100644
--- a/.github/workflows/python-tests.yml
+++ b/.github/workflows/python-tests.yml
@@ -27,8 +27,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -r requirements.txt
-        python -m pip install flake8 pytest
+        pip install -e .[dev]
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names