Repository: scrapinghub/shub

Issue:
Project-specific requirements
Would be nice to have an ability to specify project-specific requirements, so that if requirements are specified for a project then this settings overrides the global requirements file.

This feature would be very useful for developing page objects, when these're located in a separate project from Scrapy project.

The config may look like:

```yml
stacks:
  default: "scrapy:2.8"

projects:
  prod: 123456  # global requirements.txt used
  dev:
    id: 654321
    requirements:
      file: requirements-dev.txt  # project-specific requirements used

requirements:
  file: requirements.txt
```

Discussion:
codecov[bot]: ## [Codecov](https://codecov.io/gh/scrapinghub/shub/pull/426?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.01`** :tada:
> Comparison is base [(`4e85fb2`)](https://codecov.io/gh/scrapinghub/shub/commit/4e85fb2a06c7dcfbefc75b01b7cf41807c770a05?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) 89.45% compared to head [(`3f56660`)](https://codecov.io/gh/scrapinghub/shub/pull/426?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) 89.46%.

> :exclamation: Current head 3f56660 differs from pull request most recent head 3ef0148. Consider uploading reports for the commit 3ef0148 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #426      +/-   ##
==========================================
+ Coverage   89.45%   89.46%   +0.01%     
==========================================
  Files          34       34              
  Lines        2427     2430       +3     
==========================================
+ Hits         2171     2174       +3     
  Misses        256      256              
```


| [Impacted Files](https://codecov.io/gh/scrapinghub/shub/pull/426?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) | Coverage Δ | |
|---|---|---|
| [shub/config.py](https://codecov.io/gh/scrapinghub/shub/pull/426?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub#diff-c2h1Yi9jb25maWcucHk=) | `96.92% <100.00%> (+0.03%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/scrapinghub/shub/pull/426?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub).


Patch:
diff --git a/.gitignore b/.gitignore
index b411b77..d12bc36 100644
--- a/.gitignore
+++ b/.gitignore
@@ -128,3 +128,5 @@ tags
 [._]*.un~
 
 # End of https://www.gitignore.io/api/vim,python
+
+.vscode
diff --git a/docs/conf.py b/docs/conf.py
index 9a74f56..eec2595 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -30,7 +30,7 @@ VERSION = __version__.rsplit('.', 1)[0]
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-# needs_sphinx = '1.0'
+needs_sphinx = '6.0'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
@@ -308,4 +308,4 @@ if not on_rtd:  # only import and set the theme if we're building docs locally
 # Wrap text in tables
 # https://github.com/snide/sphinx_rtd_theme/issues/117#issuecomment-41571653
 def setup(app):
-    app.add_stylesheet("theme_overrides.css")
+    app.add_css_file("theme_overrides.css")
diff --git a/docs/configuration.rst b/docs/configuration.rst
index 408fa66..fce0b2b 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -81,9 +81,9 @@ The following is a list of all available configuration options:
 ================  ============================================  ===============
 Option            Description                                   Scope
 ================  ============================================  ===============
-``requirements``  Path to the project's requirements file, and  global only
-                  to any additional eggs that should be
-                  deployed to Scrapy Cloud. See
+``requirements``  Path to the project's requirements file, and  global default
+                  to any additional eggs that should be         and project-\
+                  deployed to Scrapy Cloud. See                 specific
                   :ref:`deploying-dependencies`.
 ``stack``         `Scrapy Cloud stack`_ to use (this is the     global default
                   environment that your project will run in,    and project-\
@@ -236,3 +236,20 @@ as well as different API endpoints::
     apikeys:
       default: 0bbf4f0f691e0d9378ae00ca7bcf7f0c
       vagrant: a1aeecc4cd52744730b1ea6cd3e8412a
+
+Global and project-specific requirements. ``requirements.txt`` is used for projects ``prod`` and ``some``, ``requirements-dev.txt`` and eggs for ``dev``::
+
+  projects:
+    prod: 12345
+    dev:
+        id: 345
+        requirements:
+            file: requirements-dev.txt
+            eggs:
+            - ./egg1.egg
+            - ./egg2.egg
+    some: 567
+  requirements:
+    file: requirements.txt
+  stacks:
+    default: "scrapy:2.8"
diff --git a/shub/config.py b/shub/config.py
index 150888b..6bad500 100644
--- a/shub/config.py
+++ b/shub/config.py
@@ -292,6 +292,9 @@ class ShubConfig(object):
                            "" % proj['endpoint'])
                 raise MissingAuthException(msg)
             apikey = None
+        proj_requirements = proj.get('requirements', {})
+        requirements = proj_requirements.get('file', self.requirements_file)
+        eggs = proj_requirements.get('eggs', self.eggs)
         return Target(
             project_id=proj['id'],
             endpoint=self.endpoints[proj['endpoint']],
@@ -299,9 +302,9 @@ class ShubConfig(object):
             stack=(self.stacks.get(proj['stack'], proj['stack'])
                    if 'stack' in proj else self.stacks.get('default')),
             image=self._select_image_for_project(target, proj),
-            requirements_file=self.requirements_file,
+            requirements_file=requirements,
             version=self.get_version(),
-            eggs=self.eggs,
+            eggs=eggs,
         )
 
     def _select_image_for_project(self, target, project):


Test patch:
diff --git a/tests/test_config.py b/tests/test_config.py
index 292fc47..c4c78d7 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -33,6 +33,13 @@ VALID_YAML_CFG = """
         advanced_dev:
             id: 457
             stack: dev
+        some:
+            id: 345
+            requirements:
+                file: requirements-dev.txt
+                eggs:
+                - ./egg3.egg
+                - ./egg4.egg
     endpoints:
         external: ext_endpoint
     apikeys:
@@ -109,6 +116,13 @@ class ShubConfigTest(unittest.TestCase):
                 'id': 457,
                 'stack': 'dev'
             },
+            'some': {
+                'id': 345,
+                'requirements': {
+                    'file': 'requirements-dev.txt',
+                    'eggs': ['./egg3.egg', './egg4.egg']
+                }
+            }
         }
         self.assertEqual(projects, self.conf.projects)
         endpoints = {'external': 'ext_endpoint'}
@@ -461,6 +475,14 @@ class ShubConfigTest(unittest.TestCase):
             'advanced_prod': _project_dict(
                 456, extra={'stack': 'hworker:v1.0.0'}),
             'advanced_dev': _project_dict(457, extra={'stack': 'dev'}),
+            'some': _project_dict(
+                345, extra={
+                    'requirements': {
+                        'file': 'requirements-dev.txt',
+                        'eggs': ['./egg3.egg', './egg4.egg']
+                    }
+                }
+            )
         }
         self.assertEqual(self.conf.normalized_projects, expected_projects)
 
@@ -644,6 +666,30 @@ class ShubConfigTest(unittest.TestCase):
         self.assertEqual(self.conf.get_target_conf('456'), t)
         self.assertEqual(self.conf.get_target_conf('default/456'), t)
 
+    def test_get_target_correct_requirements(self):
+        correct_reqs = {
+            'shproj': 'requirements.txt',
+            'notmeproj': 'requirements.txt',
+            'advanced_prod': 'requirements.txt',
+            'advanced_dev': 'requirements.txt',
+            'some': 'requirements-dev.txt'
+        }
+        for name, reqs_file in correct_reqs.items():
+            target = self.conf.get_target_conf(name)
+            self.assertEqual(target.requirements_file, reqs_file)
+
+    def test_get_target_correct_eggs(self):
+        correct_eggs = {
+            'shproj': ["./egg1.egg", "./egg2.egg"],
+            'notmeproj': ["./egg1.egg", "./egg2.egg"],
+            'advanced_prod': ["./egg1.egg", "./egg2.egg"],
+            'advanced_dev': ["./egg1.egg", "./egg2.egg"],
+            'some': ["./egg3.egg", "./egg4.egg"]
+        }
+        for name, eggs in correct_eggs.items():
+            target = self.conf.get_target_conf(name)
+            self.assertEqual(target.eggs, eggs)
+
     def test_get_undefined(self):
         self.assertEqual(
             self.conf.get_target_conf('99'),