Repository: sosw/sosw

Issue:
Implement param ignore_case for recursive_matches_* helpers
Default False
If True, ignore case of path elements.

Discussion:
@Praveen-kumar-Reddy-Idamakanti: for what type of data type you want ignore Case like list,dict,str etc?
@Praveen-kumar-Reddy-Idamakanti: import re 
#here we will take three arguments
#1.data, 2.pattern(pattern we are searching For), 3.Ignore Case(default to False)
def recursive_Matched(data,pattern,ignore_Case=False):
       matches=[]#for Storing Matched things
       flags=re.IGNORECASE if ignore_case else 0
       complied_Pattern=re.complie(pattern,flags)
if isinstance(data, str):
        # If data is a string, do regex matching
        if compiled_pattern.search(data):
            matches.append(data)
    elif isinstance(data, list):
        # If data is a list, recursively search each element
        for item in data:
            matches.extend(recursive_matches(item, pattern, ignore_case))
    elif isinstance(data, dict):
        # If data is a dictionary, recursively search keys and values
        for key, value in data.items():
            matches.extend(recursive_matches(key, pattern, ignore_case))
            matches.extend(recursive_matches(value, pattern, ignore_case))
return matches 



@muddi900: I can add the solution. But wouldn't it be better to pass this as a kwarg?

we can add 

```python
ignore_case = kwargs.get("ignore_case",False)
```

at the start of the method.

@ngr: I see that we already support kwargs there... Okay. then your solution is okay.

 This should be implemented in:
`components.helpers.recursive_matches_extract`

Don't forget to cover with tests. And make your
 PR not to master, but `0_7_49`
@muddi900: ```diff
:return:    Value from structure extracted by specified path
     """
 
+    ignore_case = kwargs.get("ignore_case", False)
+
+    if ignore_case:
+        key = key.lower()
+
     if any([x in kwargs for x in ['exclude_key', 'exclude_val']]) \
             and not all([x in kwargs for x in ['exclude_key', 'exclude_val']]):
```

Would this be the correct implementation?@ngr: A good start, but not enough.

I would start with writing a custom test with several test payloads, for example:
```
    def test_recursive_matches_extract__ignore_case(self):
        TESTS = [
            ({'Headers': {'Origin': 'foo'}}, 'headers.origin', 'foo'),
            ({'headers': {'oriGIN': 'foo'}}, 'Headers.OrIgiN', 'foo'),
            ...
        ]
        for payload, path, result in TESTS:
            self.assertEqual(recursive_matches_extract(payload, path, ignore_case=True), result)    
```

Note. This is just test idea, didn't check the syntax.@muddi900: The case test can't be on both the source and the search key.

that would require us to modify the payload it self.@ngr: That's exactly what we want!
Not really modify the payload, but in case this parameter is set to True, we make a deepcopy of the payload with keys lowercased just for comparison/exctraction reasons.@ngr: And since you are upgrading this function, would be nice also to support:
`({'Headers': {42: 'foo'}}, 'headers.42', 'foo')`
@muddi900: The [commit](https://github.com/muddi900/sosw/commit/b9e0bdcbbcbe3b5dd4ef4598cf693ada91ff400a) works, however it would really slow every thing if the payload is large.@muddi900: Creating exceptions for non-string keys would require a rewrite of the whole method. I believe that should be a separate issue it self.

it can be done, but the current state the function requires some maintenance. Adding more complexity can make things worse.@ngr: Almost there!!
But the lists/tuples  are failing now.

```
d = {'Headers': [{'foo': 'bar'}]}
recursive_matches_extract(d, 'Headers.foo', ignore_case=True)
...
AttributeError: 'list' object has no attribute 'items'
```

I would add the following to the beginning of your `ignore_case_copy` method:
```
if isinstance(src, (list, tuple)):
    for element in src:
        return [ignore_case_copy(element) for element in src]
```
And probably rename it to `recursive_lower_keys`, cover with a couple of simple tests and register it as a separate helper in `__all__`.

And add more complicated nested test cases please with nesting lists, dicts, etc.@ngr: Sorry. Noticed in your code another problem:
```
        if re.match(r"\d+", key):
            return src.get(key, src.get(int(key)))
```

This is a eventually bad idea.
It works for `'foo.42'` path but will break for `'foo.42.bar'` path.

Let's revert this part, because we want it to be configurable with a different kwargs parameter. And probably in a different PR with different tests.

Solution patch:
diff --git a/sosw/components/helpers.py b/sosw/components/helpers.py
index eff1b951..b9852358 100644
--- a/sosw/components/helpers.py
+++ b/sosw/components/helpers.py
@@ -593,6 +593,26 @@ def recursive_matches_strict(src, key, val, **kwargs):
         raise RuntimeError("Your function is stupid", src, key, val)
 
 
+def ignore_case_copy(src):
+    """A funtion to create a copy of a source dict with all string keys in lower case.
+    
+    Keyword arguments:
+    :param dict src -- Input dictionary
+    :return dict: a copy of the dict with all string keys in lower case.
+    """
+    
+    output = {}
+    for k, v in src.items():
+        if isinstance(k,str):
+            if isinstance(v,dict):
+                output[k.lower()] = ignore_case_copy(v)
+            output[k.lower()] = v
+            continue
+
+        output[k] = v
+    return output
+
+
 def recursive_matches_extract(src, key, separator=None, **kwargs):
     """
     Searches the 'src' recursively for nested elements provided in 'key' with dot notation.
@@ -624,6 +644,12 @@ def recursive_matches_extract(src, key, separator=None, **kwargs):
     :return:    Value from structure extracted by specified path
     """
 
+    ignore_case = kwargs.get("ignore_case", False)
+
+    if ignore_case:
+        key = key.lower()
+        src = ignore_case_copy(src) if isinstance(src,dict) else [ignore_case_copy(s) for s in src]
+
     if any([x in kwargs for x in ['exclude_key', 'exclude_val']]) \
             and not all([x in kwargs for x in ['exclude_key', 'exclude_val']]):
         raise AttributeError("If you use 'exclude' attributes you must specify both 'exclude_key' and 'exclude_val'")
@@ -657,7 +683,10 @@ def recursive_matches_extract(src, key, separator=None, **kwargs):
                 # logging.debug("Skipping element because it matches exclude parameters.")
                 return None
         except KeyError:
-            pass  # There is a chance that the exclude key is simply missing. We ignore it then.
+            pass
+        
+        
+        # There is a chance that the exclude key is simply missing. We ignore it then.
         return src.get(key)
     else:
         raise RuntimeError("Your function is stupid")


Test patch:
diff --git a/sosw/components/test/unit/test_helpers.py b/sosw/components/test/unit/test_helpers.py
index a3546bea..72d63c17 100644
--- a/sosw/components/test/unit/test_helpers.py
+++ b/sosw/components/test/unit/test_helpers.py
@@ -348,7 +348,7 @@ def test_validate_date_timestamp_from_something(self):
     def test_recursive_match_extract(self):
         SRC = {
             "bar": [{"page": {"oid": 234}}, {"page": {"code": "exclude_me", "id": 123}},
-                    {"page": {"code": "ok", "id": 333}}], "name": "test"
+                    {"page": {"code": "ok", "id": 333}}], "name": "test", "42": "foo",
         }
 
         self.assertEqual(recursive_matches_extract(SRC, 'name'), "test")
@@ -356,6 +356,7 @@ def test_recursive_match_extract(self):
         self.assertEqual(recursive_matches_extract(SRC, 'bar.page.id'), 123, "The first one in list")
         self.assertEqual(recursive_matches_extract(SRC, 'bar.page.id', exclude_key='code', exclude_val='exclude_me'),
                          333, "Exclude did not work.")
+        self.assertEqual(recursive_matches_extract(SRC, "42"), "foo")
 
         self.assertIsNone(recursive_matches_extract(SRC, 'na'))
         self.assertIsNone(recursive_matches_extract(SRC, 'bar.baz'))
@@ -365,6 +366,18 @@ def test_recursive_match_extract(self):
         self.assertRaises(AttributeError, recursive_matches_extract, SRC, 'foo.bar.baz', exclude_val=1)
 
 
+    def test_recursive_matches_extract__ignore_case(self):
+        TESTS = [
+            ({"HEADERS": {"Origin": "foo"}}, "headers.origin", "foo"),
+            ({"headers": {"oriGIN": "foo"}}, "Headers.OrIgiN", "foo"),
+            ({"hEaDeRs": {"oRiGiN": "foo"}}, "headers.origin", "foo"),
+            (({"hEaDeRs": {"oRiGiN": "foo"}},{"headers": {"origin": "foo"}}), "headers.origin", "foo")
+        ]
+
+        for payload, path, result in TESTS:
+            self.assertEqual(recursive_matches_extract(payload, path, ignore_case=True), result)
+
+
     def test_chunks(self):
         list_input = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
         list_input_2 = [[1, 2, 3], ['a'], [True, False]]