Repository: cloud-custodian/cloud-custodian

Issue:
c7n-policystream not scanning .yml files in sub-directories
Hi,

My custodian repository contains the following folder structure:

```
 directory
     |_ dev-policies
         |_ dev-policy-1.yml
         |_ dev-policy-2.yml
     |- prod-policies
         |_ prod-policy.yml
     |- baselinepolicies.yml
 
```

The command `c7n-policystream diff` output provides only policies modified in baselinepolicies.yml. It is not giving new added policies in any of the dev-policy-1.yml, dev-policy-2.yml, prod-policy.yml. Whereas, `c7n-policystream stream` displays <policy-add> in those files.

NOTE: I am running the `c7n-policystream diff` in master branch.

Thanks In Advance!


Discussion:
as noted on gitter, you'll need to provide a test git repository exhibiting the issue, stream and diff using the same underlying mechanism, and there's unit tests around the same.

fwiw, policy stream is an advanced tool, typically used by folks with significant experience around using custodian to optimize ci setups. you've mentioned your just getting started, in which case you might want to familiarize its the operation/usage basics first.
@kapilt  Here is the link to github, where issue can be reproduce. 

https://github.com/m-harini/cloud-custodian.git

Following is the output for stream and diff - 

```
$ c7n-policystream stream
<policy-add policy:account-service-limit-increase provider:aws resource:aws.account date:2020-02-20T20:35:32+05:30 author:Harini Muralidharan commit:578f1d>
<policy-add policy:Test-Custodian provider:aws resource:account date:2020-02-20T20:38:34+05:30 author:Harini Muralidharan commit:f934be>
2020-02-20 20:39:12,492: c7n.policystream:INFO Streamed 2 policy repo changes

$ c7n-policystream diff
policies: []
```
thanks for the sample repo, confirmed the issue, its a one line fix, I'll have a pr up in a bit, but would like to add a unit test case as well

```diff
@@ -287,7 +287,7 @@ class PolicyRepo(object):
         while q:
             t, prefix = q.popleft()
             for fent in t:
-                if fent.type == 'tree':
+                if fent.type == pygit2.GIT_OBJ_TREE:
```

Resolving patch:
diff --git a/tools/c7n_policystream/policystream.py b/tools/c7n_policystream/policystream.py
--- a/tools/c7n_policystream/policystream.py
+++ b/tools/c7n_policystream/policystream.py
@@ -38,7 +38,7 @@
 from c7n.credentials import SessionFactory
 from c7n.policy import PolicyCollection as BaseCollection
 from c7n.policy import Policy as BasePolicy
-from c7n.resources import load_resources
+from c7n.resources import load_available
 from c7n.utils import get_retry
 
 import boto3
@@ -287,7 +287,7 @@ def _get_policy_fents(self, tree):
         while q:
             t, prefix = q.popleft()
             for fent in t:
-                if fent.type == 'tree':
+                if fent.type == pygit2.GIT_OBJ_TREE:
                     q.append((
                         self.repo.get(fent.id),
                         os.path.join(prefix, fent.name)))
@@ -862,12 +862,12 @@ def diff(repo_uri, source, target, output, verbose):
         repo_uri = pygit2.discover_repository(os.getcwd())
 
     repo = pygit2.Repository(repo_uri)
-    load_resources()
+    load_available()
 
     # If on master show diff between last commit to current head
     if repo.head.shorthand == 'master':
         if source is None:
-            source = 'master@{1}'
+            source = 'HEAD^1'
         if target is None:
             target = 'master'
     # Else show difference between master and current head
@@ -934,7 +934,7 @@ def stream(repo_uri, stream_uri, verbose, assume, sort, before=None, after=None)
             repo = pygit2.clone_repository(repo_uri, temp_dir.path)
         else:
             repo = pygit2.Repository(repo_uri)
-        load_resources()
+        load_available()
         policy_repo = PolicyRepo(repo_uri, repo)
         change_count = 0
 


Test patch: