Repository: slacgismo/solar-data-tools (language: python)

Issue:
clear_day_detection use
Cool algorithm.  I'm looking at using in for [solarforecastarbiter](https://github.com/SolarArbiter/solarforecastarbiter-core) and have questions for you about usage: how much data is needed for a confident result?  does the time averaging interval matter?  how is the result affected by systematic shading?  what validation/reference papers could I cite?  Happy to discuss offline but I don't know how to contact you at SLAC.  Best, Cliff

Development Merg
Let's review this PR at the Monday meeting.
Mosek in the CI/CD pipeline
Added the mosek setup to the pipeline. 

Looks like the test failed after this was added though. Not sure if this is a travis thing or a mosek thing. I'm assuming the tests are passing locally for you...?


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
new file mode 100644
index 00000000..12dec740
--- /dev/null
+++ b/.github/workflows/build.yml
@@ -0,0 +1,73 @@
+name: Deploy Main
+on: 
+  push:
+    branches:
+      - master
+jobs:
+    deploy-pypi:
+        runs-on: ubuntu-latest
+        environment: deployment
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+          - name: Install Python
+            uses: actions/setup-python@v3
+            with:
+              python-version: '3.7'
+
+          - name: Install Twine
+            run: sudo pip install twine
+
+          - name: Create the distribution
+            run: |
+              git fetch --prune --unshallow --tags
+              sudo python setup.py sdist bdist_wheel
+
+          - name: Push to PyPI
+            run: sudo twine upload -u ${{ secrets.PYPI_USERNAME }} -p ${{ secrets.PYPI_PASSWORD }} dist/*
+
+    deploy-conda:
+        runs-on: ubuntu-latest
+# sets default shell to remove need for source to run the conda shell
+        defaults:
+          run:
+            shell: bash -l {0}
+        environment: deployment
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+# Much better than manual installation, original version Miniconda2-4.7.10-Linux-x86_64.sh is broken
+          - name: Install Miniconda
+            uses: conda-incubator/setup-miniconda@v2
+            with:
+              auto-activate-base: true
+              activate-environment: ""
+              miniconda-version: "latest"
+
+          - name: Install the Conda Dependencies
+            run: | 
+              conda config --set always_yes yes --set auto_update_conda false
+              conda update conda
+              conda install conda-build
+
+# run install twice due to client-size to ensure all files downloaded
+# echo yes before login to prevent anaconda bug breaking automation
+# git tags MUST be fetched otherwise output will be blank
+# bash variables cannot be used in github actions, must use actions specific syntax and methods
+          - name: Build the Anaconda Package
+            id: condabuild
+            run: |
+              conda install anaconda-client
+              conda install anaconda-client
+              conda config --set anaconda_upload no
+              echo yes | anaconda login --username ${{ secrets.ANACONDA_CLOUD_USERNAME }} --password ${{ secrets.ANACONDA_CLOUD_PASSWORD }}
+              git fetch --prune --unshallow --tags
+              VERSION_FROM_GIT_TAG=$(git tag --list "v*[0-9]" --sort=version:refname | tail -1 | cut -c 2-) conda build . -c conda-forge -c slacgismo --numpy 1.16.4
+              echo '::set-output name=gitversion::$(git tag --list "v*[0-9]" --sort=version:refname | tail -1 | cut -c 2-)'
+
+          - name: Upload the Anaconda Package
+            id: condaload
+            run: |
+              anaconda upload -u slacgismo  /usr/share/miniconda3/conda-bld/noarch/solar-data-tools-${{ steps.condabuild.outputs.gitversion }}-*.tar.bz2
\ No newline at end of file
diff --git a/.circleci/config.yml b/.oldcircle/config.yml
similarity index 100%
rename from .circleci/config.yml
rename to .oldcircle/config.yml
diff --git a/setup.py b/setup.py
index 3241169f..25202287 100644
--- a/setup.py
+++ b/setup.py
@@ -15,7 +15,7 @@
 # Python 3 only projects can skip this import
 from io import open
 
-here = Path().resolve()
+here = Path()
 
 # Get the long description from the README file
 with open((here / "README.md"), encoding="utf-8") as f:
@@ -39,7 +39,7 @@
 # PEP 440 won't accept the v in front, so here we remove it, strip the new line and decode the byte stream
 VERSION_FROM_GIT_TAG = latest_version[1:].strip().decode("utf-8")
 
-with open(os.path.join(here, "requirements.txt"), encoding="utf-8") as f:
+with open((here / "requirements.txt"), encoding="utf-8") as f:
     install_requires = f.read().splitlines()
 # removes comments in the requirements file
 dependencies = [dependency for dependency in install_requires if (dependency[0] != "#")]


Test patch:
diff --git a/.github/workflows/build-test.yml b/.github/workflows/build-test.yml
new file mode 100644
index 00000000..88b5b0b4
--- /dev/null
+++ b/.github/workflows/build-test.yml
@@ -0,0 +1,70 @@
+name: Test Deployment
+on: 
+  push:
+    branches-ignore:
+      - master
+jobs:
+    deploy-pypi:
+        runs-on: ubuntu-latest
+        environment: test-deployment
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+          - name: Install Python
+            uses: actions/setup-python@v3
+            with:
+              python-version: '3.7'
+
+          - name: Install Twine
+            run: sudo pip install twine
+
+          - name: Create the distribution
+            run: |
+              git fetch --prune --unshallow --tags
+              sudo python setup.py sdist bdist_wheel
+
+    deploy-conda:
+        runs-on: ubuntu-latest
+# sets default shell to remove need for source to run the conda shell
+        defaults:
+          run:
+            shell: bash -l {0}
+        environment: test-deployment
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+# Much better than manual installation, original version Miniconda2-4.7.10-Linux-x86_64.sh is broken
+          - name: Install Miniconda
+            uses: conda-incubator/setup-miniconda@v2
+            with:
+              auto-activate-base: true
+              activate-environment: ""
+              miniconda-version: "latest"
+
+          - name: Install the Conda Dependencies
+            run: | 
+              conda config --set always_yes yes --set auto_update_conda false
+              conda update conda
+              conda install conda-build
+
+# run install twice due to client-size to ensure all files downloaded
+# echo yes before login to prevent anaconda bug breaking automation
+# git tags MUST be fetched otherwise output will be blank
+# bash variables cannot be used in github actions, must use actions specific syntax and methods
+          - name: Build the Anaconda Package
+            id: condabuild
+            run: |
+              conda install anaconda-client
+              conda install anaconda-client
+              conda config --set anaconda_upload no
+              echo yes | anaconda login --username ${{ secrets.ANACONDA_CLOUD_USERNAME }} --password ${{ secrets.ANACONDA_CLOUD_PASSWORD }}
+              git fetch --prune --unshallow --tags
+              VERSION_FROM_GIT_TAG=$(git tag --list "v*[0-9]" --sort=version:refname | tail -1 | cut -c 2-)test conda build . -c conda-forge -c slacgismo --numpy 1.16.4
+              echo '::set-output name=gitversion::$(git tag --list "v*[0-9]" --sort=version:refname | tail -1 | cut -c 2-)'
+
+          - name: Upload the Anaconda Package
+            id: condaload
+            run: |
+              anaconda upload --force -u slacgismo  /usr/share/miniconda3/conda-bld/noarch/solar-data-tools-${{ steps.condabuild.outputs.gitversion }}test-*.tar.bz2 --label test
\ No newline at end of file
diff --git a/.github/workflows/test-pypi.yml b/.github/workflows/test-pypi.yml
new file mode 100644
index 00000000..eb0dc0c1
--- /dev/null
+++ b/.github/workflows/test-pypi.yml
@@ -0,0 +1,29 @@
+name: Test Deployment
+on: 
+  pull_request:
+    branches:
+      - master
+    types: [opened]
+jobs:
+    deploy-pypi-test:
+        runs-on: ubuntu-latest
+        environment: test-deployment
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+          - name: Install Python
+            uses: actions/setup-python@v3
+            with:
+              python-version: '3.7'
+
+          - name: Install Twine
+            run: sudo pip install twine
+
+          - name: Create the distribution
+            run: |
+              git fetch --prune --unshallow --tags
+              sudo python setup.py sdist bdist_wheel
+
+          - name: Push to PyPI
+            run: sudo twine upload -r testpypi -u ${{ secrets.PYPI_TEST_USERNAME }} -p ${{ secrets.PYPI_TEST_PASSWORD }} dist/*
\ No newline at end of file
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
new file mode 100644
index 00000000..de090546
--- /dev/null
+++ b/.github/workflows/test.yml
@@ -0,0 +1,38 @@
+name: Test Push
+on: [push]
+jobs:
+    test-build:
+        runs-on: ubuntu-latest
+        environment: test
+        steps:
+          - name: Checkout Repository
+            uses: actions/checkout@v3
+
+# Necessary action when needing to use AWS credentials.
+          - name: Configure AWS Credentials
+            uses: aws-actions/configure-aws-credentials@v1
+            with:
+              aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+              aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+              aws-region: us-west-1
+
+          - name: Install Python
+            uses: actions/setup-python@v3
+            with:
+              python-version: '3.7'
+
+          - name: Install Dependencies
+            run: | 
+              sudo pip install -r requirements.txt
+              sudo pip install awscli coverage
+
+          - name: Setup Mosek License File
+            run: |
+              sudo mkdir /root/mosek
+              mkdir $HOME/mosek
+              aws s3 cp s3://slac.gismo.ci.artifacts/mosek.license/mosek.lic $HOME/mosek/mosek.lic
+              sudo cp $HOME/mosek/mosek.lic /root/mosek/mosek.lic
+
+# Current unit test is not consistent. Occasionally fails despite usually passing. Needs to be fixed.
+          - name: Run Unit Tests
+            run: sudo coverage run -m unittest