Repository: snakemake/snakemake

Issue:
A utility function to partially format strings with wildcards
**Is your feature request related to a problem? Please describe.**
In the workflow, we usually need to deal with multiple wildcards.
It would be nice if there is a function to partially format the input string with multiple wildcards.
Maybe this function already exists and I'm just not aware of it.
For example:
```
foo = "{a}_{b}.txt"
result = format_cards(foo, a="foo")
print(result)
# result = "foo_{b}.txt"
```

**Describe the solution you'd like**
Here is a [solution](https://stackoverflow.com/questions/11283961/partial-string-formatting) I found on StackOverflow.
```
def format_cards(template, **kwargs):
    import string
    class FormatDict(dict):
        def __missing__(self, key):
        return "{" + key + "}"
    formatter = string.Formatter()
    mapping = FormatDict(**kwarg)
    return formatter.vformat(template, (), mapping)
```



Discussion:
SultanOrazbayev: This is a great feature! In terms of simplifying coding, do you think it might be better to switch to something like `from functools import partial` (see https://docs.python.org/2/library/functools.html)? This would obviate the need for the special keyword `allow_missing`.
johanneskoester: Thanks for the suggestion. I must say though that I don't see the use case. You can always mask the wildcards:
`assert expand("{a}_{{b}}.out", a=[1,2]) == ["1_{b}.out", "2_{b}.out"]`

troycomi: I typically keep my paths in the config and try to reference them in the workflow to keep everything DRY.  If my `outfile = '{a}_{b}.out'` I then have to do something like:
```python
expand(outfile.replace('b', '{b}'), a = [1, 2])
# or
expand(outfile, a=[1,2], b='{b}')
```
both obfuscate the code more and get repetitive if I have a lot of wildcards I *don't* want to format.

The other example is chaining combinators, as brought up by this [old issue](https://bitbucket.org/snakemake/snakemake/issues/1282/using-zip-for-some-but-not-all-wildcards).  The solution could now be rephrased as
```python
expand(
  expand('{tp}/compare/{name}vs{comp}', comp=sample_to_compare, allow_missing=True),
  tp=sample_name, comp=sample_to_compare, zip)
```

The [original issue](https://github.com/snakemake/snakemake/issues/124#issuecomment-558705885) also presented a use case where a partial expand would feed into listfiles or glob_wildcards.

I guess it's a convenience function for a corner case, but it would be a cleaner fix than format masking in complex examples.  It may (hopefully) prevent future issues asking for something similar.  If allow_missing isn't included in the kwargs it also doesn't add much overhead.
shenghuanjie: > Thanks for the suggestion. I must say though that I don't see the use case. You can always mask the wildcards:
> `assert expand("{a}_{{b}}.out", a=[1,2]) == ["1_{b}.out", "2_{b}.out"]`

This has also been discussed in [this](https://github.com/snakemake/snakemake/issues/124) issue.
Do you have any suggestions for a good workaround to the example mentioned in the issue?
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!

[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=BUG)  
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=SECURITY_HOTSPOT) [3 Security Hotspots](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=SECURITY_HOTSPOT) to review)  
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=snakemake_snakemake&pullRequest=127&resolved=false&types=CODE_SMELL)

[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=snakemake_snakemake&pullRequest=127) No Coverage information  
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.4%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=snakemake_snakemake&pullRequest=127&metric=new_duplicated_lines_density&view=list) [0.4% Duplication](https://sonarcloud.io/component_measures?id=snakemake_snakemake&pullRequest=127&metric=new_duplicated_lines_density&view=list)



Patch:
diff --git a/docs/project_info/faq.rst b/docs/project_info/faq.rst
index 91321e44..e1ef4d2a 100644
--- a/docs/project_info/faq.rst
+++ b/docs/project_info/faq.rst
@@ -100,6 +100,25 @@ In order to infer the IDs from present files, Snakemake provides the ``glob_wild
 
 The function matches the given pattern against the files present in the filesystem and thereby infers the values for all wildcards in the pattern. A named tuple that contains a list of values for each wildcard is returned. Here, this named tuple has only one item, that is the list of values for the wildcard ``{id}``.
 
+I don't want expand to use the product of every wildcard, what can I do?
+------------------------------------------------------------------------
+
+By default the expand function uses ``itertools.product`` to create every combination of the supplied wildcards.
+Expand takes an optional, second positional argument which can customize how wildcards are combined.
+To create the list ``["a_1.txt", "b_2.txt", "c_3.txt"]``, invoke expand as:
+``expand("{sample}_{id}.txt", zip, sample=["a", "b", "c"], id=["1", "2", "3"])``
+
+I don't want expand to use every wildcard, what can I do?
+---------------------------------------------------------
+
+Sometimes partially expanding wildcards is useful to define inputs which still depend on some wildcards.
+Expand takes an optional keyword argument, allow_missing=True, that will format only wildcards which are supplied, leaving others as is.
+To create the list ``["{sample}_1.txt", "{sample}_2.txt"]``, invoke expand as:
+``expand("{sample}_{id}.txt", id=["1", "2"], allow_missing=True)``
+If the filename contains the wildcard ``allow_missing``, it will be formatted normally:
+``expand("{allow_missing}.txt", allow_missing=True)`` returns ``["True.txt"]``.
+
+
 Snakemake complains about a cyclic dependency or a PeriodicWildcardError. What can I do?
 ----------------------------------------------------------------------------------------
 
diff --git a/snakemake/io.py b/snakemake/io.py
index fd7d5dda..a976a9e9 100755
--- a/snakemake/io.py
+++ b/snakemake/io.py
@@ -17,6 +17,7 @@ import functools
 import subprocess as sp
 from itertools import product, chain
 from contextlib import contextmanager
+import string
 import collections
 
 import yaml
@@ -892,7 +893,8 @@ def expand(*args, **wildcards):
         second arg (optional): a function to combine wildcard values
         (itertools.product per default)
     **wildcards -- the wildcards as keyword arguments
-        with their values as lists
+        with their values as lists. If allow_missing=True is included
+        wildcards in filepattern without values will stay unformatted.
     """
     filepatterns = args[0]
     if len(args) == 1:
@@ -916,12 +918,27 @@ def expand(*args, **wildcards):
             "of expand (e.g. 'temp(expand(\"plots/{sample}.pdf\", sample=SAMPLES))')."
         )
 
+    # check if remove missing is provided
+    format_dict = dict
+    if "allow_missing" in wildcards and wildcards["allow_missing"] is True:
+
+        class FormatDict(dict):
+            def __missing__(self, key):
+                return "{" + key + "}"
+
+        format_dict = FormatDict
+        # check that remove missing is not a wildcard in the filepatterns
+        for filepattern in filepatterns:
+            if "allow_missing" in re.findall(r"{([^}\.[!:]+)", filepattern):
+                format_dict = dict
+                break
+
     # remove unused wildcards to avoid duplicate filepatterns
     wildcards = {
         filepattern: {
             k: v
             for k, v in wildcards.items()
-            if k in re.findall("{([^}\.[!:]+)", filepattern)
+            if k in re.findall(r"{([^}\.[!:]+)", filepattern)
         }
         for filepattern in filepatterns
     }
@@ -934,11 +951,12 @@ def expand(*args, **wildcards):
                 values = [values]
             yield [(wildcard, value) for value in values]
 
+    formatter = string.Formatter()
     try:
         return [
-            filepattern.format(**comb)
+            formatter.vformat(filepattern, (), comb)
             for filepattern in filepatterns
-            for comb in map(dict, combinator(*flatten(wildcards[filepattern])))
+            for comb in map(format_dict, combinator(*flatten(wildcards[filepattern])))
         ]
     except KeyError as e:
         raise WildcardError("No values given for wildcard {}.".format(e))
@@ -1050,7 +1068,7 @@ def update_wildcard_constraints(
 
 
 def split_git_path(path):
-    file_sub = re.sub("^git\+file:/+", "/", path)
+    file_sub = re.sub(r"^git\+file:/+", "/", path)
     (file_path, version) = file_sub.split("@")
     file_path = os.path.realpath(file_path)
     root_path = get_git_root(file_path)


Test patch:
diff --git a/tests/test_expand.py b/tests/test_expand.py
new file mode 100644
index 00000000..8094a38b
--- /dev/null
+++ b/tests/test_expand.py
@@ -0,0 +1,67 @@
+from snakemake.io import expand
+from snakemake.exceptions import WildcardError
+import pytest
+
+
+def test_simple_expand():
+    # single filepattern
+    assert expand("{a}.out", a="test") == ["test.out"]
+    # multiple filepatterns
+    assert expand(["{a}.out", "{b}.out"], a="a", b="b") == ["a.out", "b.out"]
+    # multiple wildcards
+    assert expand("{a}.out", a=["1", "2", "3"]) == ["1.out", "2.out", "3.out"]
+    # multiple wildcards and patterns
+    assert expand(["{a}_{b}.ab", "{b}.b"], a="1 2".split(), b="3 4".split()) == [
+        "1_3.ab",
+        "1_4.ab",
+        "2_3.ab",
+        "2_4.ab",
+        "3.b",
+        "4.b",
+    ]
+    # replace product
+    assert expand(["{a}_{b}.ab", "{b}.b"], zip, a="1 2".split(), b="3 4".split()) == [
+        "1_3.ab",
+        "2_4.ab",
+        "3.b",
+        "4.b",
+    ]
+
+
+def test_allow_missing():
+    # single filepattern
+    assert expand("{a}_{b}.out", allow_missing=True) == ["{a}_{b}.out"]
+    assert expand("{a}_{b}.out", a="test", allow_missing=True) == ["test_{b}.out"]
+    # none missing
+    assert expand("{a}.out", a="test", allow_missing=True) == ["test.out"]
+    # wildcard is allow_missing
+    assert expand("{allow_missing}.out", allow_missing=True) == ["True.out"]
+    # allow_missing not True
+    assert expand("{a}.out", a="test", allow_missing="test2") == ["test.out"]
+    with pytest.raises(WildcardError) as e:
+        expand("{a}.out", allow_missing="test2")
+    assert str(e.value) == "No values given for wildcard 'a'."
+
+    # multiple filepatterns
+    assert expand(["{a}.out", "{b}.out"], allow_missing=True) == ["{a}.out", "{b}.out"]
+    # multiple wildcards
+    assert expand("{a}_{b}.out", a=["1", "2", "3"], allow_missing=True) == [
+        "1_{b}.out",
+        "2_{b}.out",
+        "3_{b}.out",
+    ]
+    # multiple wildcards and patterns
+    assert expand(
+        ["{a}_{b}_{C}.ab", "{b}_{c}.b"],
+        a="1 2".split(),
+        b="3 4".split(),
+        allow_missing=True,
+    ) == ["1_3_{C}.ab", "1_4_{C}.ab", "2_3_{C}.ab", "2_4_{C}.ab", "3_{c}.b", "4_{c}.b"]
+    # replace product
+    assert expand(
+        ["{a}_{b}_{C}.ab", "{b}_{c}.b"],
+        zip,
+        a="1 2".split(),
+        b="3 4".split(),
+        allow_missing=True,
+    ) == ["1_3_{C}.ab", "2_4_{C}.ab", "3_{c}.b", "4_{c}.b"]