Repository: jenkins-x/lighthouse
Pull request #1218 (2021-01-25 12:04:48)

fix: add support multiple git servers in uses:sourceURI
folks may want to reuse from github.com for things like tekton catalog: `uses:tektoncd/catalog/task/git-clone/0.2/git-clone.yaml@HEAD` while also using files from the current git server (if not using github.com as the git server)

so lets add an optional `uses:(lighthouse:)owner/repository/path@version` syntax to reference the lighthouse git server instead of github.com

jenkins-x-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **jstrachan**
You can assign the PR to them by writing `/assign @jstrachan` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=jenkins-x%2Flighthouse).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/jenkins-x/lighthouse/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jstrachan"]} -->
jenkins-x-bot: @jstrachan: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
jenkins | 4c46c815a87e324b18b22b2a2e232a9bcfb1e8ec | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218/9) | `/test jenkins`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jstrachan: /retest
jenkins-x-bot: @jstrachan: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
jenkins | 4c46c815a87e324b18b22b2a2e232a9bcfb1e8ec | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218/9) | `/test jenkins`
bbs | 2913dcd30d114339c46e893802a0d998d344e280 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218/3) | `/test bbs`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @jstrachan: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
jenkins | c998116685f8ef598fbef690cca7595e1f7c3757 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218/15) | `/test jenkins`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-1218) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jstrachan: /retest
rawlingsj: /lgtm
jenkins-x-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *[rawlingsj](https://github.com/jenkins-x/lighthouse/pull/1218#issuecomment-767004198 "LGTM")*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=jenkins-x%2Flighthouse).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com#issuecomment-767004198/jenkins-x/lighthouse/blob/master/OWNERS)~~ [rawlingsj]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/docs/pipelines.md b/docs/pipelines.md
index 85aeca24..249e0ddc 100644
--- a/docs/pipelines.md
+++ b/docs/pipelines.md
@@ -50,10 +50,11 @@ That is a good approach for referencing a versioned `Task` / `Pipeline` / `Pipel
 
 The source URI syntax we use is:
         
-* treat it as a URL if it can be parsed as a URL (e.g. it contains `https://` or `http://`)
+* treat it as a URL if it contains `://` 
 * if the string contains `@` then it's a git URI of the form: `owner/repository/pathToFile@versionBranchOrSha`
+  * the repository is assumed to live on `github.com` like GitHub Actions. If you wish to use your local repository prefix the sourceURI with the server name. e.g.  `lighthouse:owner/repository/pathToFile@versionBranchOrSha` will reference a git repository in the current lighthouse git repository instead of github.com
   * you can use `@HEAD` to mean the latest version from the main branch
-  * you can use `@versionStream` to mean the git SHA of this git repository configured inside your version stream if available; otherwise it defaults to `@HEAD`  
+  * you can use `@versionStream` to mean the git SHA of this git repository configured inside your version stream if available; otherwise it defaults to `@HEAD`
 * otherwise assume the path is a local relative file in git
 
 
diff --git a/pkg/filebrowser/file_browsers.go b/pkg/filebrowser/file_browsers.go
new file mode 100644
index 00000000..663901f8
--- /dev/null
+++ b/pkg/filebrowser/file_browsers.go
@@ -0,0 +1,91 @@
+package filebrowser
+
+import (
+	"os"
+	"strings"
+
+	"github.com/jenkins-x/lighthouse/pkg/git/v2"
+	"github.com/pkg/errors"
+)
+
+const (
+	// GitHub the name used in a Uses Git URI to reference a git server
+	GitHub = "github"
+
+	// Lighthouse the name used in a Uses Git URI to reference the git server lighthouse is configured to run against
+	Lighthouse = "lighthouse"
+
+	// GitHubURL the github server URL
+	GitHubURL = "https://github.com"
+)
+
+// FileBrowsers contains the file browsers for the supported git servers
+type FileBrowsers struct {
+	cache map[string]Interface
+}
+
+// NewFileBrowsers creates a new file browsers service for the lighthouse git server URL and file browser
+// if the current server URL is not github.com then creates a second file browser so that we can use the uses: Git URI
+// syntax for accessing github resources as well as to access the lighthouse git server too
+func NewFileBrowsers(serverURL string, fb Interface) (*FileBrowsers, error) {
+	if serverURL == "" {
+		serverURL = GitHubURL
+	}
+	isGitHub := strings.TrimSuffix(serverURL, "/") == GitHubURL
+
+	answer := &FileBrowsers{
+		cache: map[string]Interface{
+			Lighthouse: fb,
+		},
+	}
+
+	// lets see if we have a custom server name we want to support
+	serverName := os.Getenv("GIT_NAME")
+	if serverName != "" && serverName != GitHub {
+		answer.cache[serverName] = fb
+	}
+	var githubBrowser Interface
+	if isGitHub {
+		githubBrowser = fb
+	} else {
+		// lets create a github browser\
+		configureOpts := func(opts *git.ClientFactoryOpts) {
+			opts.Token = func() []byte {
+				return []byte(os.Getenv("GITHUB_TOKEN"))
+			}
+			opts.GitUser = func() (name, email string, err error) {
+				name = os.Getenv("GITHUB_USER_NAME")
+				email = os.Getenv("GITHUB_USER_EMAIL")
+				return
+			}
+			opts.Username = func() (login string, err error) {
+				login = os.Getenv("GITHUB_USER")
+				return
+			}
+			opts.Host = "github.com"
+			opts.Scheme = "https"
+		}
+		githubFactory, err := git.NewClientFactory(configureOpts)
+		if err != nil {
+			return nil, errors.Wrapf(err, "failed to create git client factory for github")
+		}
+		githubBrowser = NewFileBrowserFromGitClient(githubFactory)
+	}
+	answer.cache[GitHub] = githubBrowser
+	return answer, nil
+}
+
+// GitHubFileBrowser returns a git file browser for github.com
+func (f *FileBrowsers) GitHubFileBrowser() Interface {
+	return f.GetFileBrowser(GitHub)
+}
+
+// LighthouseGitFileBrowser returns the git file browser for the git server that lighthouse is configured against
+func (f *FileBrowsers) LighthouseGitFileBrowser() Interface {
+	return f.GetFileBrowser(Lighthouse)
+}
+
+// GetFileBrowser returns the file browser for the given git server name.
+func (f *FileBrowsers) GetFileBrowser(name string) Interface {
+	return f.cache[name]
+}
diff --git a/pkg/keeper/githubapp/factory.go b/pkg/keeper/githubapp/factory.go
index a32020e2..26db33da 100644
--- a/pkg/keeper/githubapp/factory.go
+++ b/pkg/keeper/githubapp/factory.go
@@ -5,6 +5,7 @@ import (
 	"github.com/jenkins-x/go-scm/scm/factory"
 	"github.com/jenkins-x/lighthouse/pkg/clients"
 	"github.com/jenkins-x/lighthouse/pkg/config"
+	"github.com/jenkins-x/lighthouse/pkg/filebrowser"
 	"github.com/jenkins-x/lighthouse/pkg/git"
 	"github.com/jenkins-x/lighthouse/pkg/keeper"
 	"github.com/jenkins-x/lighthouse/pkg/launcher"
@@ -42,11 +43,16 @@ func NewKeeperController(configAgent *config.Agent, botName string, gitKind stri
 		return []byte(gitToken)
 	})
 
+	fileBrowsers, err := filebrowser.NewFileBrowsers(serverURL, filebrowser.NewFileBrowserFromScmClient(gitproviderClient))
+	if err != nil {
+		return nil, errors.Wrapf(err, "failed to create the git file browsers")
+	}
+
 	tektonClient, _, lhClient, _, err := clients.GetAPIClients()
 	if err != nil {
 		return nil, errors.Wrap(err, "Error creating kubernetes resource clients.")
 	}
 	launcherClient := launcher.NewLauncher(lhClient, ns)
-	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, tektonClient, lhClient, ns, configAgent.Config, gitClient, maxRecordsPerPool, historyURI, statusURI, nil)
+	c, err := keeper.NewController(gitproviderClient, gitproviderClient, fileBrowsers, launcherClient, tektonClient, lhClient, ns, configAgent.Config, gitClient, maxRecordsPerPool, historyURI, statusURI, nil)
 	return c, err
 }
diff --git a/pkg/keeper/githubapp/github_app_ctrl.go b/pkg/keeper/githubapp/github_app_ctrl.go
index e161e41a..f796b948 100644
--- a/pkg/keeper/githubapp/github_app_ctrl.go
+++ b/pkg/keeper/githubapp/github_app_ctrl.go
@@ -174,7 +174,7 @@ func (g *gitHubAppKeeperController) createOwnerController(owner string, configGe
 		return nil, errors.Wrap(err, "Error creating kubernetes resource clients.")
 	}
 	launcherClient := launcher.NewLauncher(lhClient, g.ns)
-	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, tektonClient, lhClient, g.ns, configGetter, gitClient, g.maxRecordsPerPool, g.historyURI, g.statusURI, nil)
+	c, err := keeper.NewController(gitproviderClient, gitproviderClient, nil, launcherClient, tektonClient, lhClient, g.ns, configGetter, gitClient, g.maxRecordsPerPool, g.historyURI, g.statusURI, nil)
 	return c, err
 }
 
diff --git a/pkg/keeper/keeper.go b/pkg/keeper/keeper.go
index f184a36c..6a897ce8 100644
--- a/pkg/keeper/keeper.go
+++ b/pkg/keeper/keeper.go
@@ -91,6 +91,7 @@ type DefaultController struct {
 	logger         *logrus.Entry
 	config         config.Getter
 	spc            scmProviderClient
+	fileBrowsers   *filebrowser.FileBrowsers
 	launcherClient launcher
 	gc             git.Client
 	tektonClient   tektonclient.Interface
@@ -224,7 +225,7 @@ func init() {
 }
 
 // NewController makes a DefaultController out of the given clients.
-func NewController(spcSync, spcStatus *scmprovider.Client, launcherClient launcher, tektonClient tektonclient.Interface, lighthouseClient clientset.Interface, ns string, cfg config.Getter, gc git.Client, maxRecordsPerPool int, historyURI, statusURI string, logger *logrus.Entry) (*DefaultController, error) {
+func NewController(spcSync, spcStatus *scmprovider.Client, fileBrowsers *filebrowser.FileBrowsers, launcherClient launcher, tektonClient tektonclient.Interface, lighthouseClient clientset.Interface, ns string, cfg config.Getter, gc git.Client, maxRecordsPerPool int, historyURI, statusURI string, logger *logrus.Entry) (*DefaultController, error) {
 	if logger == nil {
 		logger = logrus.NewEntry(logrus.StandardLogger())
 	}
@@ -244,6 +245,7 @@ func NewController(spcSync, spcStatus *scmprovider.Client, launcherClient launch
 	return &DefaultController{
 		logger:         logger.WithField("controller", "sync"),
 		spc:            spcSync,
+		fileBrowsers:   fileBrowsers,
 		launcherClient: launcherClient,
 		tektonClient:   tektonClient,
 		lhClient:       lighthouseClient,
@@ -1348,7 +1350,8 @@ func (c *DefaultController) presubmitsByPull(sp *subpool) (map[int][]job.Presubm
 	owner := sp.org
 	repo := sp.repo
 	sharedConfig := c.config()
-	cfg, _, err := inrepo.Generate(filebrowser.NewFileBrowserFromScmClient(c.spc), sharedConfig, nil, owner, repo, "")
+	cache := inrepo.NewResolverCache()
+	cfg, _, err := inrepo.Generate(c.fileBrowsers, cache, sharedConfig, nil, owner, repo, "")
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to calculate in repo config")
 	}
diff --git a/pkg/triggerconfig/inrepo/calculate.go b/pkg/triggerconfig/inrepo/calculate.go
index 2d1a6c70..a859e64d 100644
--- a/pkg/triggerconfig/inrepo/calculate.go
+++ b/pkg/triggerconfig/inrepo/calculate.go
@@ -10,7 +10,7 @@ import (
 )
 
 // Generate generates the in repository config if enabled for this repository otherwise return the shared config
-func Generate(fileBrowser filebrowser.Interface, sharedConfig *config.Config, sharedPlugins *plugins.Configuration, owner, repo, eventRef string) (*config.Config, *plugins.Configuration, error) {
+func Generate(fileBrowsers *filebrowser.FileBrowsers, cache *ResolverCache, sharedConfig *config.Config, sharedPlugins *plugins.Configuration, owner, repo, eventRef string) (*config.Config, *plugins.Configuration, error) {
 	fullName := scm.Join(owner, repo)
 	if !sharedConfig.InRepoConfigEnabled(fullName) {
 		return sharedConfig, sharedPlugins, nil
@@ -32,13 +32,13 @@ func Generate(fileBrowser filebrowser.Interface, sharedConfig *config.Config, sh
 	}
 
 	// lets load the main branch first then merge in any changes from this PR/branch
-	refs, err := fileBrowser.GetMainAndCurrentBranchRefs(owner, repo, eventRef)
+	refs, err := fileBrowsers.LighthouseGitFileBrowser().GetMainAndCurrentBranchRefs(owner, repo, eventRef)
 	if err != nil {
 		return sharedConfig, sharedPlugins, errors.Wrapf(err, "failed to find main branch %s", fullName)
 	}
 
 	for _, ref := range refs {
-		repoConfig, _ := LoadTriggerConfig(fileBrowser, owner, repo, ref)
+		repoConfig, _ := LoadTriggerConfig(fileBrowsers, cache, owner, repo, ref)
 		if repoConfig != nil {
 			err = merge.ConfigMerge(&cfg, &pluginCfg, repoConfig, owner, repo)
 			if err != nil {
diff --git a/pkg/triggerconfig/inrepo/git_uri.go b/pkg/triggerconfig/inrepo/git_uri.go
index 84d1b0dc..ac3b3b6c 100644
--- a/pkg/triggerconfig/inrepo/git_uri.go
+++ b/pkg/triggerconfig/inrepo/git_uri.go
@@ -3,6 +3,8 @@ package inrepo
 import (
 	"strings"
 
+	"github.com/jenkins-x/lighthouse/pkg/filebrowser"
+
 	"github.com/jenkins-x/go-scm/scm"
 
 	"github.com/pkg/errors"
@@ -10,6 +12,7 @@ import (
 
 // GitURI the output of parsing Git URIs
 type GitURI struct {
+	Server     string
 	Owner      string
 	Repository string
 	Path       string
@@ -40,8 +43,17 @@ func ParseGitURI(text string) (*GitURI, error) {
 	case 3:
 		path = parts[2]
 	}
+	owner := parts[0]
+
+	server := filebrowser.GitHub
+	serverOwner := strings.SplitN(owner, ":", 2)
+	if len(serverOwner) > 1 {
+		server = serverOwner[0]
+		owner = serverOwner[1]
+	}
 	return &GitURI{
-		Owner:      parts[0],
+		Server:     server,
+		Owner:      owner,
 		Repository: parts[1],
 		Path:       path,
 		SHA:        sha,
@@ -62,5 +74,9 @@ func (u *GitURI) String() string {
 	if sha == "" {
 		sha = "head"
 	}
-	return path + "@" + sha
+	prefix := ""
+	if u.Server != "" && u.Server != filebrowser.GitHub {
+		prefix = u.Server + ":"
+	}
+	return prefix + path + "@" + sha
 }
diff --git a/pkg/triggerconfig/inrepo/load_triggers.go b/pkg/triggerconfig/inrepo/load_triggers.go
index 573ed5fa..523be194 100644
--- a/pkg/triggerconfig/inrepo/load_triggers.go
+++ b/pkg/triggerconfig/inrepo/load_triggers.go
@@ -22,8 +22,8 @@ import (
 )
 
 // MergeTriggers merges the configuration with any `lighthouse.yaml` files in the repository
-func MergeTriggers(cfg *config.Config, pluginCfg *plugins.Configuration, fileBrowser filebrowser.Interface, ownerName string, repoName string, sha string) (bool, error) {
-	repoConfig, err := LoadTriggerConfig(fileBrowser, ownerName, repoName, sha)
+func MergeTriggers(cfg *config.Config, pluginCfg *plugins.Configuration, fileBrowsers *filebrowser.FileBrowsers, cache *ResolverCache, ownerName string, repoName string, sha string) (bool, error) {
+	repoConfig, err := LoadTriggerConfig(fileBrowsers, cache, ownerName, repoName, sha)
 	if err != nil {
 		return false, errors.Wrap(err, "failed to load configs")
 	}
@@ -39,17 +39,17 @@ func MergeTriggers(cfg *config.Config, pluginCfg *plugins.Configuration, fileBro
 }
 
 // LoadTriggerConfig loads the `lighthouse.yaml` configuration files in the repository
-func LoadTriggerConfig(fileBrowser filebrowser.Interface, ownerName string, repoName string, sha string) (*triggerconfig.Config, error) {
+func LoadTriggerConfig(fileBrowsers *filebrowser.FileBrowsers, cache *ResolverCache, ownerName string, repoName string, sha string) (*triggerconfig.Config, error) {
 	m := map[string]*triggerconfig.Config{}
 	path := ".lighthouse"
-	files, err := fileBrowser.ListFiles(ownerName, repoName, path, sha)
+	files, err := fileBrowsers.LighthouseGitFileBrowser().ListFiles(ownerName, repoName, path, sha)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to find any lighthouse configuration files in repo %s/%s at sha %s", ownerName, repoName, sha)
 	}
 	for _, f := range files {
 		if isDirType(f.Type) {
 			filePath := path + "/" + f.Name + "/triggers.yaml"
-			cfg, err := loadConfigFile(fileBrowser, ownerName, repoName, filePath, sha)
+			cfg, err := loadConfigFile(fileBrowsers, cache, ownerName, repoName, filePath, sha)
 			if err != nil {
 				return nil, errors.Wrapf(err, "failed to load file %s in %s/%s with sha %s", filePath, ownerName, repoName, sha)
 			}
@@ -58,7 +58,7 @@ func LoadTriggerConfig(fileBrowser filebrowser.Interface, ownerName string, repo
 			}
 		} else if f.Name == "triggers.yaml" {
 			filePath := path + "/" + f.Name
-			cfg, err := loadConfigFile(fileBrowser, ownerName, repoName, filePath, sha)
+			cfg, err := loadConfigFile(fileBrowsers, cache, ownerName, repoName, filePath, sha)
 			if err != nil {
 				return nil, errors.Wrapf(err, "failed to load file %s in %s/%s with sha %s", filePath, ownerName, repoName, sha)
 			}
@@ -104,8 +104,8 @@ func isDirType(t string) bool {
 	return strings.ToLower(t) == "dir"
 }
 
-func loadConfigFile(client filebrowser.Interface, ownerName, repoName, path, sha string) (*triggerconfig.Config, error) {
-	data, err := client.GetFile(ownerName, repoName, path, sha)
+func loadConfigFile(fileBrowsers *filebrowser.FileBrowsers, cache *ResolverCache, ownerName, repoName, path, sha string) (*triggerconfig.Config, error) {
+	data, err := fileBrowsers.LighthouseGitFileBrowser().GetFile(ownerName, repoName, path, sha)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to find file %s in repo %s/%s with sha %s", path, ownerName, repoName, sha)
 	}
@@ -126,7 +126,7 @@ func loadConfigFile(client filebrowser.Interface, ownerName, repoName, path, sha
 			if err != nil {
 				sourcePath = filepath.Join(dir, r.SourcePath)
 			}
-			err = loadJobBaseFromSourcePath(client, &r.Base, ownerName, repoName, sourcePath, sha)
+			err = loadJobBaseFromSourcePath(fileBrowsers, cache, &r.Base, ownerName, repoName, sourcePath, sha)
 			if err != nil {
 				return nil, errors.Wrapf(err, "failed to load source for presubmit %s", r.Name)
 			}
@@ -144,7 +144,7 @@ func loadConfigFile(client filebrowser.Interface, ownerName, repoName, path, sha
 			if err != nil {
 				sourcePath = filepath.Join(dir, r.SourcePath)
 			}
-			err = loadJobBaseFromSourcePath(client, &r.Base, ownerName, repoName, sourcePath, sha)
+			err = loadJobBaseFromSourcePath(fileBrowsers, cache, &r.Base, ownerName, repoName, sourcePath, sha)
 			if err != nil {
 				return nil, errors.Wrapf(err, "failed to load source for postsubmit %s", r.Name)
 			}
@@ -156,7 +156,7 @@ func loadConfigFile(client filebrowser.Interface, ownerName, repoName, path, sha
 	return repoConfig, nil
 }
 
-func loadJobBaseFromSourcePath(client filebrowser.Interface, j *job.Base, ownerName, repoName, path, sha string) error {
+func loadJobBaseFromSourcePath(fileBrowsers *filebrowser.FileBrowsers, cache *ResolverCache, j *job.Base, ownerName, repoName, path, sha string) error {
 	var data []byte
 
 	// source path can either be a local file or a Git URL
@@ -168,7 +168,7 @@ func loadJobBaseFromSourcePath(client filebrowser.Interface, j *job.Base, ownerN
 		}
 
 	} else {
-		data, err = client.GetFile(ownerName, repoName, path, sha)
+		data, err = fileBrowsers.LighthouseGitFileBrowser().GetFile(ownerName, repoName, path, sha)
 		if err != nil {
 			return errors.Wrapf(err, "failed to find file %s in repo %s/%s with sha %s", path, ownerName, repoName, sha)
 		}
@@ -183,12 +183,13 @@ func loadJobBaseFromSourcePath(client filebrowser.Interface, j *job.Base, ownerN
 	message := fmt.Sprintf("in repo %s/%s with sha %s", ownerName, repoName, sha)
 
 	usesResolver := &UsesResolver{
-		Client:    client,
-		OwnerName: ownerName,
-		RepoName:  repoName,
-		SHA:       sha,
-		Dir:       dir,
-		Message:   message,
+		FileBrowsers: fileBrowsers,
+		Cache:        cache,
+		OwnerName:    ownerName,
+		RepoName:     repoName,
+		SHA:          sha,
+		Dir:          dir,
+		Message:      message,
 	}
 
 	prs, err := LoadTektonResourceAsPipelineRun(usesResolver, data)
diff --git a/pkg/triggerconfig/inrepo/resolver_cache.go b/pkg/triggerconfig/inrepo/resolver_cache.go
new file mode 100644
index 00000000..0b54cdff
--- /dev/null
+++ b/pkg/triggerconfig/inrepo/resolver_cache.go
@@ -0,0 +1,67 @@
+package inrepo
+
+import (
+	"sync"
+
+	tektonv1beta1 "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"
+)
+
+// ResolverCache a cache of data and pipelines to minimise
+// the git cloning with in repo configurations
+type ResolverCache struct {
+	lock          sync.RWMutex
+	pipelineCache map[string]*tektonv1beta1.PipelineRun
+	dataCache     map[string][]byte
+}
+
+// NewResolverCache creates a new resolver cache
+func NewResolverCache() *ResolverCache {
+	return &ResolverCache{
+		pipelineCache: map[string]*tektonv1beta1.PipelineRun{},
+		dataCache:     map[string][]byte{},
+	}
+}
+
+// GetData gets data from the cache if available or returns nil
+func (c *ResolverCache) GetData(sourceURI string) []byte {
+	if c == nil || sourceURI == "" {
+		return nil
+	}
+	var answer []byte
+	c.lock.Lock()
+	answer = c.dataCache[sourceURI]
+	c.lock.Unlock()
+	return answer
+}
+
+// SetData updates the cache
+func (c *ResolverCache) SetData(sourceURI string, value []byte) {
+	if c == nil || len(value) == 0 {
+		return
+	}
+	c.lock.Lock()
+	c.dataCache[sourceURI] = value
+	c.lock.Unlock()
+}
+
+// GetPipelineRun gets the PipelineRun from the cache if available or returns nil
+func (c *ResolverCache) GetPipelineRun(sourceURI string) *tektonv1beta1.PipelineRun {
+	if c == nil || sourceURI == "" {
+		return nil
+	}
+	var answer *tektonv1beta1.PipelineRun
+	c.lock.Lock()
+	answer = c.pipelineCache[sourceURI]
+	c.lock.Unlock()
+	return answer
+}
+
+// SetPipelineRun updates the cache
+func (c *ResolverCache) SetPipelineRun(sourceURI string, value *tektonv1beta1.PipelineRun) {
+	if c == nil || value == nil {
+		return
+	}
+	c.lock.Lock()
+	c.pipelineCache[sourceURI] = value
+	c.lock.Unlock()
+}
diff --git a/pkg/triggerconfig/inrepo/uses_resolver.go b/pkg/triggerconfig/inrepo/uses_resolver.go
index 37be3f44..49385c86 100644
--- a/pkg/triggerconfig/inrepo/uses_resolver.go
+++ b/pkg/triggerconfig/inrepo/uses_resolver.go
@@ -3,7 +3,6 @@ package inrepo
 import (
 	"fmt"
 	"io/ioutil"
-	"net/url"
 	"os"
 	"strings"
 
@@ -20,7 +19,8 @@ import (
 
 // UsesResolver resolves the `uses:` URI syntax
 type UsesResolver struct {
-	Client           filebrowser.Interface
+	FileBrowsers     *filebrowser.FileBrowsers
+	Cache            *ResolverCache
 	OwnerName        string
 	RepoName         string
 	SHA              string
@@ -28,8 +28,6 @@ type UsesResolver struct {
 	Message          string
 	DefaultValues    *DefaultValues
 	LocalFileResolve bool
-
-	cache map[string]*tektonv1beta1.PipelineRun
 }
 
 var (
@@ -41,10 +39,7 @@ var (
 // UsesSteps lets resolve the sourceURI to a PipelineRun and find the step or steps
 // for the given task name and/or step name then lets apply any overrides from the step
 func (r *UsesResolver) UsesSteps(sourceURI string, taskName string, step tektonv1beta1.Step) ([]tektonv1beta1.Step, error) {
-	if r.cache == nil {
-		r.cache = map[string]*tektonv1beta1.PipelineRun{}
-	}
-	pr := r.cache[sourceURI]
+	pr := r.Cache.GetPipelineRun(sourceURI)
 	if pr == nil {
 		data, err := r.GetData(sourceURI, false)
 		if err != nil {
@@ -61,7 +56,7 @@ func (r *UsesResolver) UsesSteps(sourceURI string, taskName string, step tektonv
 		if pr == nil {
 			return nil, errors.Errorf("no PipelineRun for URI %s", sourceURI)
 		}
-		r.cache[sourceURI] = pr
+		r.Cache.SetPipelineRun(sourceURI, pr)
 	}
 
 	return r.findSteps(sourceURI, pr, taskName, step)
@@ -69,10 +64,13 @@ func (r *UsesResolver) UsesSteps(sourceURI string, taskName string, step tektonv
 
 // GetData gets the data from the given source URI
 func (r *UsesResolver) GetData(path string, ignoreNotExist bool) ([]byte, error) {
-	var data []byte
-	_, err := url.ParseRequestURI(path)
-	if err == nil {
-		data, err = getPipelineFromURL(path)
+	data := r.Cache.GetData(path)
+	if len(data) > 0 {
+		return data, nil
+	}
+
+	if strings.Contains(path, "://") {
+		data, err := getPipelineFromURL(path)
 		if err != nil {
 			return nil, errors.Wrapf(err, "failed to get pipeline from URL %s ", path)
 		}
@@ -104,19 +102,33 @@ func (r *UsesResolver) GetData(path string, ignoreNotExist bool) ([]byte, error)
 		}
 		return data, nil
 	}
+	fb := r.FileBrowsers.LighthouseGitFileBrowser()
 	if gitURI != nil {
+		data := r.lookupDataCache(path)
+		if len(data) > 0 {
+			return data, nil
+		}
+
 		owner = gitURI.Owner
 		repo = gitURI.Repository
 		path = gitURI.Path
 		sha = resolveCustomSha(owner, repo, gitURI.SHA)
+
+		fb = r.FileBrowsers.GetFileBrowser(gitURI.Server)
+		if fb == nil {
+			return nil, errors.Errorf("could not find git file browser for server %s in uses: git URI %s", gitURI.Server, gitURI.String())
+		}
 	}
-	data, err = r.Client.GetFile(owner, repo, path, sha)
+	data, err = fb.GetFile(owner, repo, path, sha)
 	if err != nil && IsScmNotFound(err) {
 		err = nil
 	}
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to find file %s in repo %s/%s with sha %s", path, owner, repo, sha)
 	}
+	if gitURI != nil {
+		r.Cache.SetData(path, data)
+	}
 	return data, nil
 }
 
@@ -202,6 +214,14 @@ func (r *UsesResolver) findTaskStep(sourceURI string, task tektonv1beta1.Pipelin
 	return nil, errors.Errorf("source URI %s task %s has no step named %s", sourceURI, task.Name, name)
 }
 
+func (r *UsesResolver) lookupDataCache(path string) []byte {
+	return nil
+}
+
+func (r *UsesResolver) updateDataCache(path string, data []byte) {
+
+}
+
 // OverrideStep overrides the step with the given overrides
 func OverrideStep(step *tektonv1beta1.Step, override *tektonv1beta1.Step) {
 	if len(override.Command) > 0 {
diff --git a/pkg/webhook/create_agent.go b/pkg/webhook/create_agent.go
index ea60fb79..9dcff6ae 100644
--- a/pkg/webhook/create_agent.go
+++ b/pkg/webhook/create_agent.go
@@ -62,7 +62,8 @@ func (s *Server) CreateAgent(l *logrus.Entry, owner, repo, ref string) (plugins.
 
 func (s *Server) createAgent(pc *plugins.Agent, owner, repo, ref string) error {
 	var err error
-	pc.Config, pc.PluginConfig, err = inrepo.Generate(s.FileBrowser, pc.Config, pc.PluginConfig, owner, repo, ref)
+	cache := inrepo.NewResolverCache()
+	pc.Config, pc.PluginConfig, err = inrepo.Generate(s.FileBrowsers, cache, pc.Config, pc.PluginConfig, owner, repo, ref)
 	if err != nil {
 		return errors.Wrapf(err, "failed to calculate in repo config")
 	}
diff --git a/pkg/webhook/events.go b/pkg/webhook/events.go
index ea84dc29..0d4e556f 100644
--- a/pkg/webhook/events.go
+++ b/pkg/webhook/events.go
@@ -41,7 +41,7 @@ type Server struct {
 	ServerURL      *url.URL
 	TokenGenerator func() []byte
 	Metrics        *Metrics
-	FileBrowser    filebrowser.Interface
+	FileBrowsers   *filebrowser.FileBrowsers
 	InRepoCache    *lru.Cache
 
 	// Tracks running handlers for graceful shutdown
diff --git a/pkg/webhook/webhook.go b/pkg/webhook/webhook.go
index 59e56115..6433d839 100644
--- a/pkg/webhook/webhook.go
+++ b/pkg/webhook/webhook.go
@@ -6,7 +6,6 @@ import (
 	"io/ioutil"
 	"net/http"
 	"net/url"
-	"os"
 	"strings"
 
 	lru "github.com/hashicorp/golang-lru"
@@ -19,7 +18,6 @@ import (
 	"github.com/jenkins-x/lighthouse/pkg/launcher"
 	"github.com/jenkins-x/lighthouse/pkg/metrics"
 	"github.com/jenkins-x/lighthouse/pkg/plugins"
-	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
 	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/jenkins-x/lighthouse/pkg/version"
 	"github.com/jenkins-x/lighthouse/pkg/watcher"
@@ -210,38 +208,38 @@ func (o *WebhooksController) HandleWebhookRequests(w http.ResponseWriter, r *htt
 		LauncherClient:    o.launcher,
 	}
 
-	if o.server.FileBrowser == nil {
-		// allow an enviroment variable to enable the git based file browsing
-		if os.Getenv("FILE_BROWSER") == "git" {
-			configureOpts := func(opts *gitv2.ClientFactoryOpts) {
-				opts.Token = func() []byte {
-					return []byte(token)
-				}
-				opts.GitUser = func() (name, email string, err error) {
-					name = gitCloneUser
-					return
-				}
-				opts.Username = func() (login string, err error) {
-					login = gitCloneUser
-					return
-				}
-				if o.server.ServerURL.Host != "" {
-					opts.Host = o.server.ServerURL.Host
-				}
-				if o.server.ServerURL.Scheme != "" {
-					opts.Scheme = o.server.ServerURL.Scheme
-				}
+	if o.server.FileBrowsers == nil {
+		configureOpts := func(opts *gitv2.ClientFactoryOpts) {
+			opts.Token = func() []byte {
+				return []byte(token)
 			}
-			gitFactory, err := gitv2.NewClientFactory(configureOpts)
-			if err != nil {
-				err = errors.Wrapf(err, "failed to create git client factory for server %s", o.gitServerURL)
-				responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: %s", err.Error()))
+			opts.GitUser = func() (name, email string, err error) {
+				name = gitCloneUser
 				return
 			}
-			o.server.FileBrowser = filebrowser.NewFileBrowserFromGitClient(gitFactory)
-		} else {
-			scmProvider := scmprovider.ToClient(scmClient, o.server.ClientAgent.BotName)
-			o.server.FileBrowser = filebrowser.NewFileBrowserFromScmClient(scmProvider)
+			opts.Username = func() (login string, err error) {
+				login = gitCloneUser
+				return
+			}
+			if o.server.ServerURL.Host != "" {
+				opts.Host = o.server.ServerURL.Host
+			}
+			if o.server.ServerURL.Scheme != "" {
+				opts.Scheme = o.server.ServerURL.Scheme
+			}
+		}
+		gitFactory, err := gitv2.NewClientFactory(configureOpts)
+		if err != nil {
+			err = errors.Wrapf(err, "failed to create git client factory for server %s", o.gitServerURL)
+			responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: %s", err.Error()))
+			return
+		}
+		fb := filebrowser.NewFileBrowserFromGitClient(gitFactory)
+		o.server.FileBrowsers, err = filebrowser.NewFileBrowsers(o.gitServerURL, fb)
+		if err != nil {
+			err = errors.Wrapf(err, "failed to create git filebrowsers%s", o.gitServerURL)
+			responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: %s", err.Error()))
+			return
 		}
 	}
 


Test patch:
diff --git a/pkg/filebrowser/file_browsers_test.go b/pkg/filebrowser/file_browsers_test.go
new file mode 100644
index 00000000..c6bce98e
--- /dev/null
+++ b/pkg/filebrowser/file_browsers_test.go
@@ -0,0 +1,39 @@
+package filebrowser_test
+
+import (
+	"testing"
+
+	"github.com/jenkins-x/lighthouse/pkg/filebrowser"
+	"github.com/jenkins-x/lighthouse/pkg/git/v2"
+	"github.com/stretchr/testify/require"
+)
+
+func TestFileBrowsers(t *testing.T) {
+	cf, err := git.NewClientFactory()
+	require.NoError(t, err, "failed to create git client factory")
+
+	testCases := []struct {
+		serverURL string
+	}{
+		{
+			serverURL: "https://something.com",
+		},
+		{
+			serverURL: "https://github.com",
+		},
+		{
+			serverURL: "http://gitlab.something.com",
+		},
+	}
+
+	for _, tc := range testCases {
+		fb, err := filebrowser.NewFileBrowsers(tc.serverURL, filebrowser.NewFileBrowserFromGitClient(cf))
+		require.NoError(t, err, "failed to create file browser for  %s", tc.serverURL)
+		require.NotNil(t, fb.LighthouseGitFileBrowser(), "failed to get default file browser for server %s", tc.serverURL)
+
+		names := []string{filebrowser.Lighthouse, filebrowser.GitHub}
+		for _, name := range names {
+			require.NotNil(t, fb.GetFileBrowser(name), "failed to get file browser for server name %s", name)
+		}
+	}
+}
diff --git a/pkg/triggerconfig/inrepo/calculate_test.go b/pkg/triggerconfig/inrepo/calculate_test.go
index 07129d8e..f9ec9fb5 100644
--- a/pkg/triggerconfig/inrepo/calculate_test.go
+++ b/pkg/triggerconfig/inrepo/calculate_test.go
@@ -45,9 +45,10 @@ func TestCalculate(t *testing.T) {
 	}
 	sharedPluginConfig := &plugins.Configuration{}
 
-	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
+	fileBrowsers, err := filebrowser.NewFileBrowsers(filebrowser.GitHubURL, filebrowser.NewFileBrowserFromScmClient(scmProvider))
+	require.NoError(t, err, "failed to create filebrowsers")
 
-	cfg, pluginsCfg, err := inrepo.Generate(fileBrowser, sharedConfig, sharedPluginConfig, owner, repo, ref)
+	cfg, pluginsCfg, err := inrepo.Generate(fileBrowsers, inrepo.NewResolverCache(), sharedConfig, sharedPluginConfig, owner, repo, ref)
 	require.NoError(t, err, "failed to calculate in repo config")
 
 	require.NoError(t, err, "failed to invoke getClientAndTrigger")
@@ -124,8 +125,10 @@ func TestTriggersInBranchMergeToMaster(t *testing.T) {
 	}
 	sharedPluginConfig := &plugins.Configuration{}
 
-	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
-	cfg, _, err := inrepo.Generate(fileBrowser, sharedConfig, sharedPluginConfig, owner, repo, ref)
+	fileBrowsers, err := filebrowser.NewFileBrowsers(filebrowser.GitHubURL, filebrowser.NewFileBrowserFromScmClient(scmProvider))
+	require.NoError(t, err, "failed to create filebrowsers")
+
+	cfg, _, err := inrepo.Generate(fileBrowsers, inrepo.NewResolverCache(), sharedConfig, sharedPluginConfig, owner, repo, ref)
 	require.NoError(t, err, "failed to calculate in repo config")
 
 	presubmits := cfg.Presubmits[fullName]
diff --git a/pkg/triggerconfig/inrepo/git_uri_test.go b/pkg/triggerconfig/inrepo/git_uri_test.go
index df1ca021..0291f927 100644
--- a/pkg/triggerconfig/inrepo/git_uri_test.go
+++ b/pkg/triggerconfig/inrepo/git_uri_test.go
@@ -19,6 +19,7 @@ func TestParseGitURI(t *testing.T) {
 		{
 			text: "myowner/myrepo@v1",
 			expected: &inrepo.GitURI{
+				Server:     "github",
 				Owner:      "myowner",
 				Repository: "myrepo",
 				Path:       "",
@@ -28,6 +29,7 @@ func TestParseGitURI(t *testing.T) {
 		{
 			text: "myowner/myrepo/@v1",
 			expected: &inrepo.GitURI{
+				Server:     "github",
 				Owner:      "myowner",
 				Repository: "myrepo",
 				Path:       "",
@@ -38,6 +40,7 @@ func TestParseGitURI(t *testing.T) {
 		{
 			text: "myowner/myrepo/myfile.yaml@v1",
 			expected: &inrepo.GitURI{
+				Server:     "github",
 				Owner:      "myowner",
 				Repository: "myrepo",
 				Path:       "myfile.yaml",
@@ -47,6 +50,17 @@ func TestParseGitURI(t *testing.T) {
 		{
 			text: "myowner/myrepo/javascript/pullrequest.yaml@v1",
 			expected: &inrepo.GitURI{
+				Server:     "github",
+				Owner:      "myowner",
+				Repository: "myrepo",
+				Path:       "javascript/pullrequest.yaml",
+				SHA:        "v1",
+			},
+		},
+		{
+			text: "myserver:myowner/myrepo/javascript/pullrequest.yaml@v1",
+			expected: &inrepo.GitURI{
+				Server:     "myserver",
 				Owner:      "myowner",
 				Repository: "myrepo",
 				Path:       "javascript/pullrequest.yaml",
diff --git a/pkg/triggerconfig/inrepo/load_integration_test.go b/pkg/triggerconfig/inrepo/load_integration_test.go
index 253310c2..ec6c8a49 100644
--- a/pkg/triggerconfig/inrepo/load_integration_test.go
+++ b/pkg/triggerconfig/inrepo/load_integration_test.go
@@ -29,8 +29,10 @@ func TestMergeConfigIntegration(t *testing.T) {
 	cfg := &config.Config{}
 	pluginCfg := &plugins.Configuration{}
 
-	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
-	flag, err := MergeTriggers(cfg, pluginCfg, fileBrowser, repoOwner, repoName, sha)
+	fileBrowsers, err := filebrowser.NewFileBrowsers(filebrowser.GitHubURL, filebrowser.NewFileBrowserFromScmClient(scmProvider))
+	require.NoError(t, err, "failed to create filebrowsers")
+
+	flag, err := MergeTriggers(cfg, pluginCfg, fileBrowsers, NewResolverCache(), repoOwner, repoName, sha)
 	require.NoError(t, err, "failed to merge configs")
 	assert.True(t, flag, "did not return merge flag")
 
diff --git a/pkg/triggerconfig/inrepo/load_pipelinerun_test.go b/pkg/triggerconfig/inrepo/load_pipelinerun_test.go
index e9d9f5aa..139f18a0 100644
--- a/pkg/triggerconfig/inrepo/load_pipelinerun_test.go
+++ b/pkg/triggerconfig/inrepo/load_pipelinerun_test.go
@@ -18,7 +18,7 @@ import (
 
 var (
 	// generateTestOutput enable to regenerate the expected output
-	generateTestOutput = true
+	generateTestOutput = false
 )
 
 func TestLoadPipelineRunTest(t *testing.T) {
@@ -29,11 +29,7 @@ func TestLoadPipelineRunTest(t *testing.T) {
 	scmClient, _ := fake.NewDefault()
 	scmProvider := scmprovider.ToClient(scmClient, "my-bot")
 
-	resolver := &UsesResolver{
-		Client:           filebrowser.NewFileBrowserFromScmClient(scmProvider),
-		OwnerName:        "myorg",
-		LocalFileResolve: true,
-	}
+	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
 
 	// lets use a custom version stream sha
 	os.Setenv("LIGHTHOUSE_VERSIONSTREAM_JENKINS_X_JX3_PIPELINE_CATALOG", "myversionstreamref")
@@ -52,6 +48,20 @@ func TestLoadPipelineRunTest(t *testing.T) {
 			t.Logf("ignoring test %s\n", name)
 			continue
 		}
+
+		sourceURL := filebrowser.GitHubURL
+		if name == "uses-steps-custom-git" {
+			sourceURL = "https://my.gitserver.com"
+		}
+		fileBrowsers, err := filebrowser.NewFileBrowsers(sourceURL, fileBrowser)
+		require.NoError(t, err, "failed to create filebrowsers")
+
+		resolver := &UsesResolver{
+			FileBrowsers:     fileBrowsers,
+			OwnerName:        "myorg",
+			LocalFileResolve: true,
+		}
+
 		dir := filepath.Join(sourceDir, name)
 		resolver.Dir = dir
 
diff --git a/pkg/triggerconfig/inrepo/load_triggers_test.go b/pkg/triggerconfig/inrepo/load_triggers_test.go
index 42775035..e973bff1 100644
--- a/pkg/triggerconfig/inrepo/load_triggers_test.go
+++ b/pkg/triggerconfig/inrepo/load_triggers_test.go
@@ -35,10 +35,12 @@ func TestMergeConfig(t *testing.T) {
 		},
 	}
 
+	fileBrowsers, err := filebrowser.NewFileBrowsers(filebrowser.GitHubURL, filebrowser.NewFileBrowserFromScmClient(scmProvider))
+	require.NoError(t, err, "failed to create filebrowsers")
+
 	cfg := &config.Config{}
 	pluginCfg := &plugins.Configuration{}
-	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
-	flag, err := MergeTriggers(cfg, pluginCfg, fileBrowser, owner, repo, ref)
+	flag, err := MergeTriggers(cfg, pluginCfg, fileBrowsers, NewResolverCache(), owner, repo, ref)
 	require.NoError(t, err, "failed to merge configs")
 	assert.True(t, flag, "did not return merge flag")
 
@@ -52,13 +54,15 @@ func TestMergeConfig(t *testing.T) {
 func TestInvalidConfigs(t *testing.T) {
 	scmClient, _ := fake.NewDefault()
 	scmProvider := scmprovider.ToClient(scmClient, "my-bot")
-	fileBrowser := filebrowser.NewFileBrowserFromScmClient(scmProvider)
+
+	fileBrowsers, err := filebrowser.NewFileBrowsers(filebrowser.GitHubURL, filebrowser.NewFileBrowserFromScmClient(scmProvider))
+	require.NoError(t, err, "failed to create filebrowsers")
 
 	invalidRepos := []string{"duplicate-presubmit", "duplicate-postsubmit"}
 	for _, repo := range invalidRepos {
 		owner := "myorg"
 		ref := "master"
-		_, err := LoadTriggerConfig(fileBrowser, owner, repo, ref)
+		_, err := LoadTriggerConfig(fileBrowsers, NewResolverCache(), owner, repo, ref)
 		require.Errorf(t, err, "should have failed to load triggers from repo %s/%s with ref %s", owner, repo, ref)
 
 		t.Logf("got expected error loading invalid configuration on repo %s of: %s", repo, err.Error())
@@ -75,7 +79,7 @@ func TestLoadJobFromURL(t *testing.T) {
 		File("test_data/load_url/foo.yaml")
 
 	j := &job.Base{}
-	err := loadJobBaseFromSourcePath(nil, j, "", "", "https://raw.githubusercontent.com/rawlingsj/test/master/foo.yaml", "")
+	err := loadJobBaseFromSourcePath(nil, NewResolverCache(), j, "", "", "https://raw.githubusercontent.com/rawlingsj/test/master/foo.yaml", "")
 	assert.NoError(t, err, "should not have an error returned")
 	assert.Equal(t, "jenkinsxio/chuck:0.0.1", j.PipelineRunSpec.PipelineSpec.Tasks[0].TaskSpec.Steps[0].Image, "image name for task is not correct")
 }
diff --git a/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/README.md b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/README.md
new file mode 100644
index 00000000..3ac1aa9e
--- /dev/null
+++ b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/README.md
@@ -0,0 +1,3 @@
+## Reuse steps in a task 
+
+This example shows how we can use `image: uses:sourceURI` and a `name: mystep` to include individual the steps in task using both both github repository and the local lighthouse git server
\ No newline at end of file
diff --git a/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/expected.yaml b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/expected.yaml
new file mode 100644
index 00000000..3dac3705
--- /dev/null
+++ b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/expected.yaml
@@ -0,0 +1,241 @@
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+metadata:
+  creationTimestamp: null
+spec:
+  pipelineSpec:
+    params:
+    - description: the unique build number
+      name: BUILD_ID
+      type: string
+    - description: the name of the job which is the trigger context name
+      name: JOB_NAME
+      type: string
+    - description: the specification of the job
+      name: JOB_SPEC
+      type: string
+    - description: '''the kind of job: postsubmit or presubmit'''
+      name: JOB_TYPE
+      type: string
+    - description: the base git reference of the pull request
+      name: PULL_BASE_REF
+      type: string
+    - description: the git sha of the base of the pull request
+      name: PULL_BASE_SHA
+      type: string
+    - default: ""
+      description: git pull request number
+      name: PULL_NUMBER
+      type: string
+    - default: ""
+      description: git pull request ref in the form 'refs/pull/$PULL_NUMBER/head'
+      name: PULL_PULL_REF
+      type: string
+    - default: ""
+      description: git revision to checkout (branch, tag, sha, ref…)
+      name: PULL_PULL_SHA
+      type: string
+    - description: git pull reference strings of base and latest in the form 'master:$PULL_BASE_SHA,$PULL_NUMBER:$PULL_PULL_SHA:refs/pull/$PULL_NUMBER/head'
+      name: PULL_REFS
+      type: string
+    - description: git repository name
+      name: REPO_NAME
+      type: string
+    - description: git repository owner (user or organisation)
+      name: REPO_OWNER
+      type: string
+    - description: git url to clone
+      name: REPO_URL
+      type: string
+    tasks:
+    - name: from-build-pack
+      params:
+      - name: BUILD_ID
+        value: $(params.BUILD_ID)
+      - name: JOB_NAME
+        value: $(params.JOB_NAME)
+      - name: JOB_SPEC
+        value: $(params.JOB_SPEC)
+      - name: JOB_TYPE
+        value: $(params.JOB_TYPE)
+      - name: PULL_BASE_REF
+        value: $(params.PULL_BASE_REF)
+      - name: PULL_BASE_SHA
+        value: $(params.PULL_BASE_SHA)
+      - name: PULL_NUMBER
+        value: $(params.PULL_NUMBER)
+      - name: PULL_PULL_REF
+        value: $(params.PULL_PULL_REF)
+      - name: PULL_PULL_SHA
+        value: $(params.PULL_PULL_SHA)
+      - name: PULL_REFS
+        value: $(params.PULL_REFS)
+      - name: REPO_NAME
+        value: $(params.REPO_NAME)
+      - name: REPO_OWNER
+        value: $(params.REPO_OWNER)
+      - name: REPO_URL
+        value: $(params.REPO_URL)
+      taskSpec:
+        metadata: {}
+        params:
+        - description: the unique build number
+          name: BUILD_ID
+          type: string
+        - description: the name of the job which is the trigger context name
+          name: JOB_NAME
+          type: string
+        - description: the specification of the job
+          name: JOB_SPEC
+          type: string
+        - description: '''the kind of job: postsubmit or presubmit'''
+          name: JOB_TYPE
+          type: string
+        - description: the base git reference of the pull request
+          name: PULL_BASE_REF
+          type: string
+        - description: the git sha of the base of the pull request
+          name: PULL_BASE_SHA
+          type: string
+        - default: ""
+          description: git pull request number
+          name: PULL_NUMBER
+          type: string
+        - default: ""
+          description: git pull request ref in the form 'refs/pull/$PULL_NUMBER/head'
+          name: PULL_PULL_REF
+          type: string
+        - default: ""
+          description: git revision to checkout (branch, tag, sha, ref…)
+          name: PULL_PULL_SHA
+          type: string
+        - description: git pull reference strings of base and latest in the form 'master:$PULL_BASE_SHA,$PULL_NUMBER:$PULL_PULL_SHA:refs/pull/$PULL_NUMBER/head'
+          name: PULL_REFS
+          type: string
+        - description: git repository name
+          name: REPO_NAME
+          type: string
+        - description: git repository owner (user or organisation)
+          name: REPO_OWNER
+          type: string
+        - description: git url to clone
+          name: REPO_URL
+          type: string
+        stepTemplate:
+          env:
+          - name: BUILD_ID
+            value: $(params.BUILD_ID)
+          - name: JOB_NAME
+            value: $(params.JOB_NAME)
+          - name: JOB_SPEC
+            value: $(params.JOB_SPEC)
+          - name: JOB_TYPE
+            value: $(params.JOB_TYPE)
+          - name: PULL_BASE_REF
+            value: $(params.PULL_BASE_REF)
+          - name: PULL_BASE_SHA
+            value: $(params.PULL_BASE_SHA)
+          - name: PULL_NUMBER
+            value: $(params.PULL_NUMBER)
+          - name: PULL_PULL_REF
+            value: $(params.PULL_PULL_REF)
+          - name: PULL_PULL_SHA
+            value: $(params.PULL_PULL_SHA)
+          - name: PULL_REFS
+            value: $(params.PULL_REFS)
+          - name: REPO_NAME
+            value: $(params.REPO_NAME)
+          - name: REPO_OWNER
+            value: $(params.REPO_OWNER)
+          - name: REPO_URL
+            value: $(params.REPO_URL)
+          name: ""
+          resources: {}
+        steps:
+        - image: $(params.gitInitImage)
+          name: clone
+          resources: {}
+          script: |
+            #!/bin/sh
+            set -eu -o pipefail
+
+            if [[ "$(params.verbose)" == "true" ]] ; then
+              set -x
+            fi
+
+            CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
+
+            cleandir() {
+              # Delete any existing contents of the repo directory if it exists.
+              #
+              # We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
+              # or the root of a mounted volume.
+              if [[ -d "$CHECKOUT_DIR" ]] ; then
+                # Delete non-hidden files and directories
+                rm -rf "$CHECKOUT_DIR"/*
+                # Delete files and directories starting with . but excluding ..
+                rm -rf "$CHECKOUT_DIR"/.[!.]*
+                # Delete files and directories starting with .. plus any other character
+                rm -rf "$CHECKOUT_DIR"/..?*
+              fi
+            }
+
+            if [[ "$(params.deleteExisting)" == "true" ]] ; then
+              cleandir
+            fi
+
+            test -z "$(params.httpProxy)" || export HTTP_PROXY=$(params.httpProxy)
+            test -z "$(params.httpsProxy)" || export HTTPS_PROXY=$(params.httpsProxy)
+            test -z "$(params.noProxy)" || export NO_PROXY=$(params.noProxy)
+
+            /ko-app/git-init \
+              -url "$(params.url)" \
+              -revision "$(params.revision)" \
+              -refspec "$(params.refspec)" \
+              -path "$CHECKOUT_DIR" \
+              -sslVerify="$(params.sslVerify)" \
+              -submodules="$(params.submodules)" \
+              -depth "$(params.depth)"
+            cd "$CHECKOUT_DIR"
+            RESULT_SHA="$(git rev-parse HEAD)"
+            EXIT_CODE="$?"
+            if [ "$EXIT_CODE" != 0 ] ; then
+              exit $EXIT_CODE
+            fi
+            # ensure we don't add a trailing newline to the result
+            echo -n "$RESULT_SHA" > $(results.commit.path)
+            echo -n "$(params.url)" > $(results.url.path)
+        - image: gcr.io/jenkinsxio/jx-boot:3.1.155
+          name: jx-variables
+          resources: {}
+          script: |
+            #!/usr/bin/env sh
+            jx gitops variables
+        - image: node:12-slim
+          name: build-npm-install
+          resources: {}
+          script: |
+            #!/bin/sh
+            npm install
+        - image: node:12-slim
+          name: build-npm-test
+          resources: {}
+          script: |
+            #!/bin/sh
+            CI=true DISPLAY=:99 npm test
+        - image: gcr.io/kaniko-project/executor:debug-v1.3.0
+          name: build-container-build
+          resources: {}
+          script: |
+            #!/busybox/sh
+            source .jx/variables.sh
+            cp /tekton/creds-secrets/tekton-container-registry-auth/.dockerconfigjson /kaniko/.docker/config.json
+            /kaniko/executor $KANIKO_FLAGS --context=/workspace/source --dockerfile=Dockerfile --destination=$DOCKER_REGISTRY/$DOCKER_REGISTRY_ORG/$APP_NAME:$VERSION
+        - image: gcr.io/jenkinsxio/jx-preview:0.0.143
+          name: promote-jx-preview
+          resources: {}
+          script: |
+            #!/usr/bin/env sh
+            source .jx/variables.sh
+            jx preview create
+status: {}
diff --git a/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/source.yaml b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/source.yaml
new file mode 100644
index 00000000..ab54bca2
--- /dev/null
+++ b/pkg/triggerconfig/inrepo/test_data/load_pipelinerun/uses-steps-custom-git/source.yaml
@@ -0,0 +1,17 @@
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+spec:
+  pipelineSpec:
+    tasks:
+    - name: from-build-pack
+      taskSpec:
+        stepTemplate:
+          image: uses:lighthouse:jenkins-x/jx3-pipeline-catalog/packs/javascript/.lighthouse/jenkins-x/pullrequest.yaml@1d39235ee9235d7d52d4025a8e59cb8bda04306a
+        steps:
+        - name: clone
+          image: uses:tektoncd/catalog/task/git-clone/0.2/git-clone.yaml@HEAD
+        - name: jx-variables
+        - name: build-npm-install
+        - name: build-npm-test
+        - name: build-container-build
+        - name: promote-jx-preview