Repository: conan-io/conan

Issue:
Enhancement: propogate source file info in package_info
There are at least three known use cases for propagating information about sources of a package to downstream dependencies.  This includes: 

1. Debugging - Its highly probable that step-through debugging into upstream sources in Conan cache will be possible once generators have access to those source paths. 
1. Code Generation - Many projects and frameworks generate sources intended to be compiled directly into downstream projects. 
1. source_requires - Within Boost, my organization, and likely in other OSS frameworks, there are libraries which crossed some boundaries in terms of "dependency management best-practices", and directly include or depend on one or more source files of other libraries.  

This request is divided into two parts:  `package_info()` and `source_requires()`

Part 1:  `package_info()`
The first part of this request is to first add the following two members and a helper functions for use within the `conanfile.py` `package_info()` method:
```
self.cpp_info.srcdirs = "src"
self.cpp_info.srcs.append(tools.collect_srcs(self, folder))
self.cpp_info.srcs.append(tools.collect_srcs(self, folder))
```

It seems easy to model after the behavior for propagating library information, and could have similar default logic for collecting common/known file extensions, and searching any directories appended to `srcdirs`.  This part seems very straightforward to implement. 

Part 2: `source_requires()`
The second part of this request can come later, and is more difficult to think about and less clear to implement.  It is to enable downstream packages to depend purely on the sources of upstreams, as is needed for several libraries in Boost.  We have an manually implemented example here: 

https://github.com/bincrafters/conan-boost_fiber/blob/testing/1.67.0/conanfile.py#L15
https://github.com/bincrafters/conan-boost_base/blob/testing/1.67.0/conanfile.py#L124

There are several nuances and design questions to be decided before implementing a general case, but hopefully these examples make the intention clear, and can guide future discussion. 

Ping @memsharded per his request. 

- [X] I've read the [CONTRIBUTING guide](https://raw.githubusercontent.com/conan-io/conan/develop/.github/CONTRIBUTING.md).
- [X ] I've specified the Conan version, operating system version and any tool that can be relevant.
- [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.




Discussion:
There is also a missing part to be resolved, the source is not packaged (in most cases it shouldn't) but I agree from the consumer packages, for the mentioned reasons, you need the source code. So, how and when the "source()"  is fired to retrieve the source code of a dependency tree? It could be an explicit command from the consumer. 

But if you declare the `cpp_info.srcs` and there is no downloaded source code from your dependency tree it could be very confusing. I would like to see an error saying something like "You are accessing to `cpp_info.srcs` folders but the source code is not available. Call "conan xxx" to retrieve it" but I don't know how could be implemented.
The problem with ``source()`` is that it doesn't necessarily account for all sources, as there could be generated code in ``build()``, or like the ``config.h`` is different. So, letting users package the final sources together with the binary packages,  and giving a mechanism to address that, seems what this request is about.
I understand that packaging sources is "safer" than retrieve them on demand, but I don't see very reasonable to store the sources in every package. Also, a package generated without sources cannot be debugged by a consumer. So the `generated code in build(), or like the config.h` sounds like a corner case to be solved in a different way, maybe mixing both approaches.
We already capture sources along with headers and binaries in each package for a few reasons, and we do need to capture separately for each binary. That's not really a question honestly.  For "Part 1" , the only thing missing is the dedicated fields for listing them and propagating sources and source directories downstream. That's the immediate request. 
I also understand that for you it is not really a question. But for the general use, I think it is.
I resist thinking that the only way to have the sources is copying them in every binary package.
How did you justify it for `include` directories? 

Of note, my original proposal for this last year suggested maintaining the sources in a special place to maximize efficiency.  

I concluded that it was better to package redundantly, and left it out this time for a number of reasons.  First, it substantially complicates the implementation of the feature, to the point that it might get stalled because it adds so much logic.  Second, `include` directories are captured redundantly today, and if it hasn't been a problem for all this time, it should be fine for sources as well.  Third, because in retrospect, it actually is better.  I think it's a big advantage that the `build` directory contains all the files from the project which are necessary to `build` it, and the `package` directory contains all the files from the project which are necessary to "consume" it.  If you put sources in a completely different place, it seems worse.



Resolving patch:
diff --git a/conans/client/generators/cmake.py b/conans/client/generators/cmake.py
--- a/conans/client/generators/cmake.py
+++ b/conans/client/generators/cmake.py
@@ -1,15 +1,16 @@
-from conans.model import Generator
-from conans.paths import BUILD_INFO_CMAKE
 from conans.client.generators.cmake_common import cmake_dependency_vars,\
     cmake_macros, generate_targets_section, cmake_dependencies, cmake_package_info,\
     cmake_global_vars, cmake_user_info_vars, cmake_settings_info
+from conans.model import Generator
+from conans.paths import BUILD_INFO_CMAKE
 
 
 class DepsCppCmake(object):
     def __init__(self, cpp_info):
         def join_paths(paths):
             # Paths are doubled quoted, and escaped (but spaces)
-            return "\n\t\t\t".join('"%s"' % p.replace('\\', '/').replace('$', '\\$').replace('"', '\\"')
+            return "\n\t\t\t".join('"%s"'
+                                   % p.replace('\\', '/').replace('$', '\\$').replace('"', '\\"')
                                    for p in paths)
 
         def join_flags(separator, values):
@@ -28,6 +29,7 @@ def join_defines(values, prefix=""):
         self.res_paths = join_paths(cpp_info.res_paths)
         self.bin_paths = join_paths(cpp_info.bin_paths)
         self.build_paths = join_paths(cpp_info.build_paths)
+        self.src_paths = join_paths(cpp_info.src_paths)
 
         self.libs = join_flags(" ", cpp_info.libs)
         self.defines = join_defines(cpp_info.defines, "-D")
diff --git a/conans/client/generators/cmake_common.py b/conans/client/generators/cmake_common.py
--- a/conans/client/generators/cmake_common.py
+++ b/conans/client/generators/cmake_common.py
@@ -3,6 +3,7 @@
 set(CONAN_LIB_DIRS_{dep}{build_type} {deps.lib_paths})
 set(CONAN_BIN_DIRS_{dep}{build_type} {deps.bin_paths})
 set(CONAN_RES_DIRS_{dep}{build_type} {deps.res_paths})
+set(CONAN_SRC_DIRS_{dep}{build_type} {deps.src_paths})
 set(CONAN_BUILD_DIRS_{dep}{build_type} {deps.build_paths})
 set(CONAN_LIBS_{dep}{build_type} {deps.libs})
 set(CONAN_DEFINES_{dep}{build_type} {deps.defines})
@@ -41,7 +42,8 @@ def cmake_user_info_vars(deps_user_info):
     lines = []
     for dep, the_vars in deps_user_info.items():
         for name, value in the_vars.vars.items():
-            lines.append('set(CONAN_USER_%s_%s %s)' % (dep.upper(), name, _cmake_string_representation(value)))
+            lines.append('set(CONAN_USER_%s_%s %s)'
+                         % (dep.upper(), name, _cmake_string_representation(value)))
     return "\n".join(lines)
 
 
@@ -64,7 +66,8 @@ def cmake_settings_info(settings):
     for item in settings.items():
         key, value = item
         name = "CONAN_SETTINGS_%s" % key.upper().replace(".", "_")
-        settings_info += "set({key} {value})\n".format(key=name, value=_cmake_string_representation(value))
+        settings_info += "set({key} {value})\n".format(key=name,
+                                                       value=_cmake_string_representation(value))
     return settings_info
 
 
diff --git a/conans/model/build_info.py b/conans/model/build_info.py
--- a/conans/model/build_info.py
+++ b/conans/model/build_info.py
@@ -16,6 +16,7 @@ class _CppInfo(object):
     """
     def __init__(self):
         self.includedirs = []  # Ordered list of include paths
+        self.srcdirs = []  # Ordered list of source paths
         self.libdirs = []  # Directories to find libraries
         self.resdirs = []  # Directories to find resources, data, etc
         self.bindirs = []  # Directories to find executables and shared libs
@@ -34,6 +35,7 @@ def __init__(self):
         self._bin_paths = None
         self._build_paths = None
         self._res_paths = None
+        self._src_paths = None
         self.version = None  # Version of the conan package
         self.description = None  # Description of the conan package
 
@@ -54,6 +56,12 @@ def lib_paths(self):
             self._lib_paths = self._filter_paths(self.libdirs)
         return self._lib_paths
 
+    @property
+    def src_paths(self):
+        if self._src_paths is None:
+            self._src_paths = self._filter_paths(self.srcdirs)
+        return self._src_paths
+
     @property
     def bin_paths(self):
         if self._bin_paths is None:
@@ -117,6 +125,7 @@ def merge_lists(seq1, seq2):
             return [s for s in seq1 if s not in seq2] + seq2
 
         self.includedirs = merge_lists(self.includedirs, dep_cpp_info.include_paths)
+        self.srcdirs = merge_lists(self.srcdirs, dep_cpp_info.src_paths)
         self.libdirs = merge_lists(self.libdirs, dep_cpp_info.lib_paths)
         self.bindirs = merge_lists(self.bindirs, dep_cpp_info.bin_paths)
         self.resdirs = merge_lists(self.resdirs, dep_cpp_info.res_paths)
@@ -142,6 +151,10 @@ def include_paths(self):
     def lib_paths(self):
         return self.libdirs
 
+    @property
+    def src_paths(self):
+        return self.srcdirs
+
     @property
     def bin_paths(self):
         return self.bindirs


Test patch:
diff --git a/conans/test/generators/cmake_test.py b/conans/test/generators/cmake_test.py
--- a/conans/test/generators/cmake_test.py
+++ b/conans/test/generators/cmake_test.py
@@ -1,18 +1,18 @@
+import os
 import re
 import unittest
-from collections import namedtuple
 
-from conans.client.generators.cmake_multi import CMakeMultiGenerator
-from conans.model.settings import Settings
-from conans.model.conan_file import ConanFile
+from collections import namedtuple
+from conans.client.conf import default_settings_yml
 from conans.client.generators.cmake import CMakeGenerator
+from conans.client.generators.cmake_multi import CMakeMultiGenerator
 from conans.model.build_info import CppInfo
+from conans.model.conan_file import ConanFile
+from conans.model.env_info import EnvValues
 from conans.model.ref import ConanFileReference
-from conans.client.conf import default_settings_yml
+from conans.model.settings import Settings
 from conans.test.utils.test_files import temp_folder
 from conans.util.files import save
-import os
-from conans.model.env_info import EnvValues
 
 
 class CMakeGeneratorTest(unittest.TestCase):
@@ -70,9 +70,8 @@ def paths_cmake_multi_user_vars_test(self):
     def paths_cmake_test(self):
         settings_mock = namedtuple("Settings", "build_type, os, os_build, constraint, items")
         conanfile = ConanFile(None, None)
-        conanfile.initialize(settings_mock(None, None, None,
-                                                        lambda x, raise_undefined_field: x,
-                                                        lambda: {}), EnvValues())
+        conanfile.initialize(settings_mock(None, None, None, lambda x, raise_undefined_field: x,
+                                           lambda: {}), EnvValues())
         ref = ConanFileReference.loads("MyPkg/0.1@lasote/stables")
         tmp_folder = temp_folder()
         save(os.path.join(tmp_folder, "lib", "mylib.lib"), "")
diff --git a/conans/test/generators/generators_test.py b/conans/test/generators/generators_test.py
--- a/conans/test/generators/generators_test.py
+++ b/conans/test/generators/generators_test.py
@@ -1,8 +1,11 @@
 import os
 import platform
 import unittest
+
 from conans.test.utils.tools import TestClient
 from conans.util.files import load
+from conans.model.ref import ConanFileReference
+import re
 
 
 class GeneratorsTest(unittest.TestCase):
@@ -38,18 +41,39 @@ def test_base(self):
         client.save(files)
         client.run("install . --build")
 
-        virtualenvFiles = ["activate.sh", "deactivate.sh"]
+        venv_files = ["activate.sh", "deactivate.sh"]
         if platform.system() == "Windows":
-            virtualenvFiles.extend(["activate.bat", "deactivate.bat", "activate.ps1",
-                                    "deactivate.ps1"])
+            venv_files.extend(["activate.bat", "deactivate.bat", "activate.ps1",
+                               "deactivate.ps1"])
 
         self.assertEqual(sorted(['conanfile.txt', 'conaninfo.txt', 'conanbuildinfo.cmake',
                                  'conanbuildinfo.gcc', 'conanbuildinfo.qbs', 'conanbuildinfo.pri',
                                  'SConscript_conan', 'conanbuildinfo.txt', 'conanbuildinfo.props',
                                  'conanbuildinfo.vsprops', 'conanbuildinfo.xcconfig',
-                                 'conan_ycm_flags.json', 'conan_ycm_extra_conf.py'] + virtualenvFiles),
+                                 'conan_ycm_flags.json', 'conan_ycm_extra_conf.py'] + venv_files),
                          sorted(os.listdir(client.current_folder)))
 
+    def test_srcdirs(self):
+        client = TestClient()
+        conanfile = """from conans import ConanFile
+from conans.tools import save
+import os
+class TestConan(ConanFile):
+    def package(self):
+        save(os.path.join(self.package_folder, "src/file.h"), "//header")
+    def package_info(self):
+        self.cpp_info.srcdirs = ["src"]
+"""
+
+        client.save({"conanfile.py": conanfile})
+        client.run("create . mysrc/0.1@user/testing")
+        client.run("install mysrc/0.1@user/testing -g cmake")
+
+        cmake = load(os.path.join(client.current_folder, "conanbuildinfo.cmake"))
+        src_dirs = re.search('set\(CONAN_SRC_DIRS_MYSRC "(.*)"\)', cmake).group(1)
+        self.assertIn("mysrc/0.1/user/testing/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/src",
+                      src_dirs)
+
     def test_qmake(self):
         client = TestClient()
         dep = """
diff --git a/conans/test/integration/cmake_flags_test.py b/conans/test/integration/cmake_flags_test.py
--- a/conans/test/integration/cmake_flags_test.py
+++ b/conans/test/integration/cmake_flags_test.py
@@ -1,11 +1,12 @@
+import os
 import platform
 import unittest
-import os
+
 from nose.plugins.attrib import attr
 from parameterized.parameterized import parameterized
 
-from conans.model.version import Version
 from conans.client.build.cmake import CMake
+from conans.model.version import Version
 from conans.test.utils.tools import TestClient