Repository: infinitered/solidarity
Pull request #248 (2019-09-05 22:20:44)

--fix flag
This extends rules with a fix property and adds a fix option to the solidarity command that allows users to run the specified fix script when a rule fails. Fixes could be anything from installing a binary via brew to opening documentation in a browser.

codecov[bot]: # [Codecov](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=h1) Report
> Merging [#248](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=desc) into [master](https://codecov.io/gh/infinitered/solidarity/commit/f742bbe24e748bb0df50a5848b5312ac591f9e4e?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `62.5%`.

[![Impacted file tree graph](https://codecov.io/gh/infinitered/solidarity/pull/248/graphs/tree.svg?width=650&token=eTjcyCooFB&height=150&src=pr)](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #248      +/-   ##
==========================================
- Coverage   80.91%   80.91%   -0.01%     
==========================================
  Files          35       36       +1     
  Lines         697      702       +5     
  Branches      133      136       +3     
==========================================
+ Hits          564      568       +4     
  Misses         88       88              
- Partials       45       46       +1
```


| [Impacted Files](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/types.ts](https://codecov.io/gh/infinitered/solidarity/pull/248/diff?src=pr&el=tree#diff-c3JjL3R5cGVzLnRz) | `100% <ø> (ø)` | :arrow_up: |
| [src/extensions/functions/setOutputMode.ts](https://codecov.io/gh/infinitered/solidarity/pull/248/diff?src=pr&el=tree#diff-c3JjL2V4dGVuc2lvbnMvZnVuY3Rpb25zL3NldE91dHB1dE1vZGUudHM=) | `40% <40%> (ø)` | |
| [src/commands/solidarity.ts](https://codecov.io/gh/infinitered/solidarity/pull/248/diff?src=pr&el=tree#diff-c3JjL2NvbW1hbmRzL3NvbGlkYXJpdHkudHM=) | `51.21% <66.66%> (+2.19%)` | :arrow_up: |
| [src/extensions/functions/checkRequirement.ts](https://codecov.io/gh/infinitered/solidarity/pull/248/diff?src=pr&el=tree#diff-c3JjL2V4dGVuc2lvbnMvZnVuY3Rpb25zL2NoZWNrUmVxdWlyZW1lbnQudHM=) | `77.41% <81.81%> (+0.22%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=footer). Last update [f742bbe...9e4ddf0](https://codecov.io/gh/infinitered/solidarity/pull/248?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

jamonholmgren: This is awesome!
tabrindle: I really like this, and would love to see where this feature could go! As a sister project to envinfo, I've always envisioned a set of install/uninstall scripts as a library to help "fix" solidarity issues. Maybe in the future we could provide some of these scripts, and offer the custom script option we have here!
GantMan: Ryan, do you wanna touch up that message and then we can merge?  
ryanlntn: @GantMan I updated the error message and confirmed that the require vs. import is no longer needed.

Patch:
diff --git a/docs/cliOptions.md b/docs/cliOptions.md
index a45fa30..681679c 100644
--- a/docs/cliOptions.md
+++ b/docs/cliOptions.md
@@ -21,6 +21,7 @@ Solidarity
   --solidarityFile	 (-f) Use given path to solidarity file for settings
   --module		 (-d) Search for a solidarity file in the given npm package
   --stack		 (-t) Use a known technology stack, and not the local file
+  --fix		     Apply fixes to failing rules if fix is available on rule
 
 Solidarity is open source - https://github.com/infinitered/solidarity
 If you need additional help, join our Slack at http://community.infinite.red
@@ -53,3 +54,6 @@ Passing `--stack` or `-t` flags will make our stack look to GitHub for a well kn
 > For example: `solidarity --stack react-native` will check our machine if we are ready to run React Native projects, but not a specific React Native project.
 
 Stacks are community managed and found here: https://github.com/infinitered/solidarity-stacks
+
+## fix
+Passing `--fix` flag will run fix scripts on any failing rules providing them.
\ No newline at end of file
diff --git a/docs/options.md b/docs/options.md
index 7b4f0ad..a4dbfdd 100644
--- a/docs/options.md
+++ b/docs/options.md
@@ -144,3 +144,19 @@ _e.g._ Rule only performs a check on Mac and Linux
     }
   ]
 ```
+
+### Rule Fixes
+
+Many times rules can be fixed in an automated manor. You can supply a script by passing the `"fix"` property on any rule. The fix will only be ran if the `--fix` flag is passed and only against failing rules. Any script errors will surface should it fail.
+
+_e.g._ Rule provides fix to install yarn if not found (only ran if passing `--fix` option)
+
+```json
+  "Yarn": [
+    {
+      "rule": "cli",
+      "binary": "yarn",
+      "fix": "brew install yarn"
+    }
+  ]
+```
\ No newline at end of file
diff --git a/solidaritySchema.json b/solidaritySchema.json
index 41a108f..b511b83 100644
--- a/solidaritySchema.json
+++ b/solidaritySchema.json
@@ -50,7 +50,8 @@
         "matchIndex": { "type": "integer" },
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string"},
-        "ci": { "type": "boolean" }
+        "ci": { "type": "boolean" },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "binary"]
     },
@@ -61,7 +62,8 @@
         "rule": { "enum": [ "dir", "directory" ] },
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string"},
-        "ci": { "type": "boolean" }
+        "ci": { "type": "boolean" },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "location"]
     },
@@ -72,7 +74,8 @@
         "rule": { "enum": [ "file" ] },
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string"},
-        "ci": { "type": "boolean" }
+        "ci": { "type": "boolean" },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "location"]
     },
@@ -83,7 +86,8 @@
         "rule": { "enum": [ "env" ] },
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string"},
-        "ci": { "type": "boolean" }
+        "ci": { "type": "boolean" },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "variable"]
     },
@@ -95,7 +99,8 @@
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string" },
         "ci": { "type": "boolean" },
-        "match": { "type": "string", "description": "A regexp to search the output." }
+        "match": { "type": "string", "description": "A regexp to search the output." },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "match"]
     },
@@ -110,7 +115,8 @@
         "platform": { "enum": ["darwin", "macos", "freebsd", "linux", "sunos", "win32", "windows"] },
         "error": { "type": "string" },
         "ci": { "type": "boolean" },
-        "match": { "type": "string", "description": "A regexp to search the output." }
+        "match": { "type": "string", "description": "A regexp to search the output." },
+        "fix": { "type": "string"}
       },
       "required": ["rule", "plugin", "name"]
     }
diff --git a/src/commands/fix.ts b/src/commands/fix.ts
new file mode 100644
index 0000000..b7cfee7
--- /dev/null
+++ b/src/commands/fix.ts
@@ -0,0 +1,68 @@
+import { GluegunCommand } from 'gluegun'
+import { SolidarityOutputMode, SolidarityRequirementChunk, SolidarityRunContext } from '../types'
+import Listr from 'listr'
+
+module.exports = {
+  alias: 'f',
+  description: 'Applies all specified fixes for rules',
+  run: async (context: SolidarityRunContext) => {
+    // Node Modules Quirk
+    require('../extensions/functions/quirksNodeModules')
+
+    const { toPairs } = require('ramda')
+    const { print, solidarity } = context
+    const { checkRequirement, getSolidaritySettings, setOutputMode } = solidarity
+
+    // get settings or error
+    let solidaritySettings
+    try {
+      solidaritySettings = await getSolidaritySettings(context)
+    } catch (e) {
+      print.error(e.message || 'No Solidarity Settings Found')
+      print.info(
+        `Make sure you are in the correct folder or run ${print.colors.success(
+          'solidarity onboard'
+        )} to create a .solidarity file for this project.`
+      )
+      process.exit(3)
+    }
+
+    // Merge flags and configs
+    context.outputMode = setOutputMode(context.parameters, solidaritySettings)
+    // Adjust output depending on mode
+    let listrSettings: Object = { concurrent: true, collapse: false, exitOnError: false }
+    switch (context.outputMode) {
+      case SolidarityOutputMode.SILENT:
+        listrSettings = { ...listrSettings, renderer: 'silent' }
+        break
+      case SolidarityOutputMode.MODERATE:
+        // have input clear itself
+        listrSettings = { ...listrSettings, clearOutput: true }
+    }
+
+    // build Listr of checks
+    const checks = new Listr(
+      await toPairs(solidaritySettings.requirements).map((requirement: SolidarityRequirementChunk) => ({
+        title: requirement[0],
+        task: async () => checkRequirement(requirement, context, true),
+      })),
+      listrSettings
+    )
+
+    // run the array of promises in Listr
+    await checks
+      .run()
+      .then(results => {
+        const silentOutput = context.outputMode === SolidarityOutputMode.SILENT
+        // Add empty line between final result if printing rule results
+        if (!silentOutput) print.success('')
+        if (!silentOutput) print.success(print.checkmark + ' Solidarity checks valid')
+      })
+      .catch(_err => {
+        const silentOutput = context.outputMode === SolidarityOutputMode.SILENT
+        // Used to have message in the err, but that goes away with `exitOnError: false` so here's a generic one
+        if (!silentOutput) print.error('Solidarity checks failed')
+        process.exit(2)
+      })
+  },
+} as GluegunCommand
diff --git a/src/commands/solidarity.ts b/src/commands/solidarity.ts
index 9ca8827..20503a4 100644
--- a/src/commands/solidarity.ts
+++ b/src/commands/solidarity.ts
@@ -1,5 +1,5 @@
 import { GluegunCommand } from 'gluegun'
-import { SolidarityRequirementChunk, SolidarityOutputMode, SolidaritySettings, SolidarityRunContext } from '../types'
+import { SolidarityRequirementChunk, SolidarityOutputMode, SolidarityRunContext } from '../types'
 // Have to do this for tests rather than import
 const Listr = require('listr')
 
@@ -21,30 +21,14 @@ namespace Solidarity {
     }
   }
 
-  const setOutputMode = (parameters, settings: SolidaritySettings): SolidarityOutputMode => {
-    const { options } = parameters
-    // CLI flags override config
-    if (options.verbose || options.a) {
-      return SolidarityOutputMode.VERBOSE
-    } else if (options.silent || options.s) {
-      return SolidarityOutputMode.SILENT
-    } else if (options.moderate || options.m) {
-      return SolidarityOutputMode.MODERATE
-    }
-
-    // Set output mode, set to default on invalid value
-    let outputModeString = settings.config ? String(settings.config.output).toUpperCase() : 'MODERATE'
-    return SolidarityOutputMode[outputModeString] || SolidarityOutputMode.MODERATE
-  }
-
   export const run = async (context: SolidarityRunContext) => {
     // Node Modules Quirk
     require('../extensions/functions/quirksNodeModules')
     // drop out fast in these situations
     await checkForEscapeHatchFlags(context)
 
-    const { print, solidarity } = context
-    const { checkRequirement, getSolidaritySettings } = solidarity
+    const { parameters, print, solidarity } = context
+    const { checkRequirement, getSolidaritySettings, setOutputMode } = solidarity
 
     // get settings or error
     let solidaritySettings
@@ -77,7 +61,7 @@ namespace Solidarity {
     const checks = new Listr(
       await toPairs(solidaritySettings.requirements).map((requirement: SolidarityRequirementChunk) => ({
         title: requirement[0],
-        task: async () => checkRequirement(requirement, context),
+        task: async () => checkRequirement(requirement, context, Boolean(parameters.options.fix)),
       })),
       listrSettings
     )
diff --git a/src/extensions/functions/checkRequirement.ts b/src/extensions/functions/checkRequirement.ts
index 8e677bd..0bed8e3 100644
--- a/src/extensions/functions/checkRequirement.ts
+++ b/src/extensions/functions/checkRequirement.ts
@@ -1,7 +1,14 @@
-import { SolidarityRule, SolidarityRequirementChunk, SolidarityRequirement, SolidarityRunContext } from '../../types'
+import {
+  SolidarityRule,
+  SolidarityRequirementChunk,
+  SolidarityRequirement,
+  SolidarityRunContext,
+  SolidarityChecker,
+} from '../../types'
 module.exports = async (
   requirement: SolidarityRequirementChunk,
-  context: SolidarityRunContext
+  context: SolidarityRunContext,
+  shouldRunFix = false
 ): Promise<void | object[]> => {
   const checkCLI = require('./checkCLI')
   const checkENV = require('./checkENV')
@@ -22,6 +29,21 @@ module.exports = async (
     flatten
   )(requirement)
 
+  const taskWithFix = (checker: SolidarityChecker, rule: SolidarityRule, context: SolidarityRunContext) => async () => {
+    if (!shouldRunFix) return checker(rule, context)
+    try {
+      const result = await checker(rule, context)
+      return result
+    } catch (error) {
+      if (rule.fix) {
+        await context.system.run(rule.fix)
+        return checker(rule, context)
+      } else {
+        throw new Error('No fix script provided in .solidarity file')
+      }
+    }
+  }
+
   const configureSubtask = rule => {
     let subTask: Object = {}
     switch (rule.rule) {
@@ -33,7 +55,7 @@ module.exports = async (
         subTask = {
           title: ruleString,
           skip: () => skipRule(rule),
-          task: async () => checkCLI(rule, context),
+          task: taskWithFix(checkCLI, rule, context),
         }
         break
       // Handle ENV rule check
@@ -41,7 +63,7 @@ module.exports = async (
         subTask = {
           title: `${rule.variable} env`,
           skip: () => skipRule(rule),
-          task: async () => checkENV(rule, context),
+          task: taskWithFix(checkENV, rule, context),
         }
         break
       // Handle dir rule check
@@ -50,7 +72,7 @@ module.exports = async (
         subTask = {
           title: `${rule.location} directory exists`,
           skip: () => skipRule(rule),
-          task: async () => checkDir(rule, context),
+          task: taskWithFix(checkDir, rule, context),
         }
         break
       // Handle file rule check
@@ -58,7 +80,7 @@ module.exports = async (
         subTask = {
           title: `${rule.location} file exists`,
           skip: () => skipRule(rule),
-          task: async () => checkFile(rule, context),
+          task: taskWithFix(checkFile, rule, context),
         }
         break
       // Handle the shell rule
@@ -66,7 +88,7 @@ module.exports = async (
         subTask = {
           title: `'${rule.command}' matches '${rule.match}'`,
           skip: () => skipRule(rule),
-          task: async () => checkShell(rule, context),
+          task: taskWithFix(checkShell, rule, context),
         }
         break
       case 'custom':
@@ -77,7 +99,7 @@ module.exports = async (
             // takes into account they didn't provide a check
             skip: () => skipRule(rule) || !customPluginRule.plugin.check,
             task: async () => {
-              const customResult = await customPluginRule.plugin.check(rule, context)
+              const customResult = await taskWithFix(customPluginRule.plugin.check, rule, context)()
               if (customResult && customResult.pass) {
                 return true
               } else {
diff --git a/src/extensions/functions/setOutputMode.ts b/src/extensions/functions/setOutputMode.ts
new file mode 100644
index 0000000..278e25c
--- /dev/null
+++ b/src/extensions/functions/setOutputMode.ts
@@ -0,0 +1,17 @@
+import { SolidarityOutputMode, SolidaritySettings } from '../../types'
+
+module.exports = (parameters, settings: SolidaritySettings): SolidarityOutputMode => {
+  const { options } = parameters
+  // CLI flags override config
+  if (options.verbose || options.a) {
+    return SolidarityOutputMode.VERBOSE
+  } else if (options.silent || options.s) {
+    return SolidarityOutputMode.SILENT
+  } else if (options.moderate || options.m) {
+    return SolidarityOutputMode.MODERATE
+  }
+
+  // Set output mode, set to default on invalid value
+  let outputModeString = settings.config ? String(settings.config.output).toUpperCase() : 'MODERATE'
+  return SolidarityOutputMode[outputModeString] || SolidarityOutputMode.MODERATE
+}
diff --git a/src/types.ts b/src/types.ts
index 8a0c0ca..8fd20c6 100644
--- a/src/types.ts
+++ b/src/types.ts
@@ -15,6 +15,7 @@ export const solidarity = {
   binaryExists: require('./extensions/functions/binaryExists'),
   getSolidaritySettings: require('./extensions/functions/getSolidaritySettings'),
   setSolidaritySettings: require('./extensions/functions/setSolidaritySettings'),
+  setOutputMode: require('./extensions/functions/setOutputMode'),
   checkRequirement: require('./extensions/functions/checkRequirement'),
   updateRequirement: require('./extensions/functions/updateRequirement'),
   skipRule: require('./extensions/functions/skipRule'),
@@ -72,6 +73,7 @@ export interface CLIRule {
   readonly matchIndex?: number
   platform?: string | string[]
   ci?: boolean // updatable
+  fix?: string
 }
 
 export interface ENVRule {
@@ -80,6 +82,7 @@ export interface ENVRule {
   error?: string
   platform?: string | string[]
   ci?: boolean
+  fix?: string
 }
 
 export interface FSRule {
@@ -88,6 +91,7 @@ export interface FSRule {
   error?: string
   platform?: string | string[]
   ci?: boolean
+  fix?: string
 }
 
 /**
@@ -121,6 +125,7 @@ export interface ShellRule {
    * An optional flag to skip rule on CI
    */
   ci?: boolean
+  fix?: string
 }
 
 export interface CustomRule {
@@ -130,11 +135,14 @@ export interface CustomRule {
   error?: string
   platform?: string | string[]
   ci?: boolean
+  fix?: string
 }
 
 // discriminated union for rule sets
 export type SolidarityRule = CLIRule | ENVRule | FSRule | ShellRule | CustomRule
 
+export type SolidarityChecker = (SolidarityRule, SolidarityRunContext) => any
+
 export interface PluginFind {
   success: boolean
   message?: string


Test patch:
diff --git a/__tests__/command_helpers/checkRequirement.ts b/__tests__/command_helpers/checkRequirement.ts
index a686054..3aa907c 100644
--- a/__tests__/command_helpers/checkRequirement.ts
+++ b/__tests__/command_helpers/checkRequirement.ts
@@ -48,6 +48,7 @@ describe('checkRequirement', () => {
     }
     context.strings = strings
     context.system = {
+      run: jest.fn().mockResolvedValue(true),
       spawn: jest.fn().mockReturnValue(
         Promise.resolve({
           stdout: 'you did it!',
@@ -61,6 +62,51 @@ describe('checkRequirement', () => {
     await expect(checkRequirement(badRule, context)).rejects.toThrow()
   })
 
+  describe('when fix option is passed', () => {
+    beforeEach(() => checkCLI.mockClear())
+
+    test('when no fix is provided', async () => {
+      checkCLI.mockImplementation(async () => {
+        throw new Error("Binary 'yarn' not found")
+      })
+
+      const rule = toPairs({
+        YARN: [{ rule: 'cli', binary: 'yarn' }],
+      })[0]
+      const listrTask = await checkRequirement(rule, context, true)
+
+      await expect(listrTask.storedInit[0].task()).rejects.toThrow(
+        new Error('No fix script provided in .solidarity file')
+      )
+    })
+
+    test('when a fix should be applied', async () => {
+      checkCLI.mockImplementation(async () => {
+        throw new Error("Binary 'yarn' not found")
+      })
+
+      const rule = toPairs({
+        YARN: [{ rule: 'cli', binary: 'yarn', fix: 'brew install yarn' }],
+      })[0]
+      const listrTask = await checkRequirement(rule, context, true)
+
+      await expect(listrTask.storedInit[0].task()).rejects.toThrow()
+      expect(context.system.run).toHaveBeenCalledWith('brew install yarn')
+    })
+
+    test('when a fix should not be applied', async () => {
+      checkCLI.mockImplementation(async () => false)
+
+      const rule = toPairs({
+        YARN: [{ rule: 'cli', binary: 'yarn', fix: 'brew install yarn' }],
+      })[0]
+      const listrTask = await checkRequirement(rule, context, true)
+      await listrTask.storedInit[0].task()
+
+      expect(context.system.run).not.toHaveBeenCalled()
+    })
+  })
+
   describe('when rule: cli', () => {
     beforeEach(() => checkCLI.mockClear())
 
@@ -123,15 +169,17 @@ describe('checkRequirement', () => {
       expect(result).toEqual('It worked!')
     })
 
-    // test('sad path', async () => {
-    //   checkDir.mockImplementation(() => throw new Error('Nope'))
+    test('sad path', async () => {
+      checkDir.mockImplementation(() => {
+        throw new Error('Nope')
+      })
 
-    //   const rule = toPairs({
-    //     YARN: [{ rule: 'dir', location: 'yarn' }],
-    //   })[0]
-    //   const listrTask = await checkRequirement(rule, context)
-    //   await expect(listrTask.storedInit[0].task()).rejects.toThrow()
-    // })
+      const rule = toPairs({
+        YARN: [{ rule: 'dir', location: 'yarn' }],
+      })[0]
+      const listrTask = await checkRequirement(rule, context)
+      await expect(listrTask.storedInit[0].task()).rejects.toThrow()
+    })
   })
 
   describe('when rule: env', () => {
@@ -150,16 +198,16 @@ describe('checkRequirement', () => {
       expect(runResult).toEqual(false)
     })
 
-    // test('sad path', async () => {
-    //   checkENV.mockImplementation(async () => undefined)
+    test('sad path', async () => {
+      checkENV.mockImplementation(async () => undefined)
 
-    //   const rule = toPairs({
-    //     YARN: [{ rule: 'env', variable: 'yarn' }],
-    //   })[0]
-    //   const listrTask = await checkRequirement(rule, context)
-    //   const result = await listrTask.storedInit[0].task()
-    //   expect(result).toEqual(undefined)
-    // })
+      const rule = toPairs({
+        YARN: [{ rule: 'env', variable: 'yarn' }],
+      })[0]
+      const listrTask = await checkRequirement(rule, context)
+      const result = await listrTask.storedInit[0].task()
+      expect(result).toEqual(undefined)
+    })
   })
 
   describe('when rule: file', () => {
@@ -188,60 +236,60 @@ describe('checkRequirement', () => {
     // })
   })
 
-  // describe('when rule fails with custom error messages', () => {
-  //   const customError = 'customError'
-
-  //   test('failed CLI rule with custom message', async () => {
-  //     checkCLI.mockClear()
-  //     checkCLI.mockImplementation(() => customError)
-  //     const rule = toPairs({
-  //       YARN: [{ rule: 'cli', binary: 'gazorpazorp', error: customError }],
-  //     })[0]
-
-  //     const listrTask = await checkRequirement(rule, context)
-  //     const result = await listrTask.storedInit[0].task()
-  //     expect(result).toEqual(customError)
-  //   })
-
-  //   test('failed ENV rule with custom message', async () => {
-  //     checkCLI.mockClear()
-  //     checkCLI.mockImplementation(() => true)
-  //     const rule = toPairs({
-  //       BADENV: [{ rule: 'env', variable: 'gazorpazorp', error: customError }],
-  //     })[0]
-
-  //     const listrTask = await checkRequirement(rule, context)
-  //     const result = await listrTask.storedInit[0].task()
-  //     expect(result).toEqual(undefined)
-  //   })
-
-  //   test('failed DIR rule with custom message', async () => {
-  //     const rule = toPairs({
-  //       YARN: [{ rule: 'dir', location: 'gazorpazorp', error: customError }],
-  //     })[0]
-
-  //     const listrTask = await checkRequirement(rule, context)
-  //     const result = await listrTask.storedInit[0].task()
-  //     expect(result).toEqual(false)
-  //   })
-
-  // test('failed FILE rule with custom message', async () => {
-  //   const rule = toPairs({
-  //     YARN: [{ rule: 'file', location: 'gazorpazorp', error: customError }],
-  //   })[0]
-
-  //   const listrTask = await checkRequirement(rule, context)
-  //   const result = await listrTask.storedInit[0].task()
-  //   expect(result).toEqual(undefined)
-  // })
-
-  // test('failed SHELL rule with custom message', async () => {
-  //   const rule = toPairs({
-  //     YARN: [{ rule: 'shell', match: 'hello', command: 'mocked', error: customError }],
-  //   })[0]
-  //   const listrTask = await checkRequirement(rule, context)
-  //   const result = await listrTask.storedInit[0].task()
-  //   expect(result).toEqual(customError)
-  // })
-  // })
+  describe('when rule fails with custom error messages', () => {
+    const customError = 'customError'
+
+    test('failed CLI rule with custom message', async () => {
+      checkCLI.mockClear()
+      checkCLI.mockImplementation(() => customError)
+      const rule = toPairs({
+        YARN: [{ rule: 'cli', binary: 'gazorpazorp', error: customError }],
+      })[0]
+
+      const listrTask = await checkRequirement(rule, context)
+      const result = await listrTask.storedInit[0].task()
+      expect(result).toEqual(customError)
+    })
+
+    test('failed ENV rule with custom message', async () => {
+      checkCLI.mockClear()
+      checkCLI.mockImplementation(() => true)
+      const rule = toPairs({
+        BADENV: [{ rule: 'env', variable: 'gazorpazorp', error: customError }],
+      })[0]
+
+      const listrTask = await checkRequirement(rule, context)
+      const result = await listrTask.storedInit[0].task()
+      expect(result).toEqual(undefined)
+    })
+
+    // test('failed DIR rule with custom message', async () => {
+    //   const rule = toPairs({
+    //     YARN: [{ rule: 'dir', location: 'gazorpazorp', error: customError }],
+    //   })[0]
+
+    //   const listrTask = await checkRequirement(rule, context)
+    //   const result = await listrTask.storedInit[0].task()
+    //   expect(result).toEqual(false)
+    // })
+
+    // test('failed FILE rule with custom message', async () => {
+    //   const rule = toPairs({
+    //     YARN: [{ rule: 'file', location: 'gazorpazorp', error: customError }],
+    //   })[0]
+
+    //   const listrTask = await checkRequirement(rule, context)
+    //   const result = await listrTask.storedInit[0].task()
+    //   expect(result).toEqual(undefined)
+    // })
+
+    // test('failed SHELL rule with custom message', async () => {
+    //   const rule = toPairs({
+    //     YARN: [{ rule: 'shell', match: 'hello', command: 'mocked', error: customError }],
+    //   })[0]
+    //   const listrTask = await checkRequirement(rule, context)
+    //   const result = await listrTask.storedInit[0].task()
+    //   expect(result).toEqual(customError)
+    // })
+  })
 })
diff --git a/__tests__/command_helpers/setOutputMode.ts b/__tests__/command_helpers/setOutputMode.ts
new file mode 100644
index 0000000..b467030
--- /dev/null
+++ b/__tests__/command_helpers/setOutputMode.ts
@@ -0,0 +1,42 @@
+import { SolidarityOutputMode } from '../../src/types'
+import setOutputMode from '../../src/extensions/functions/setOutputMode'
+
+test('default output mode', () => {
+  const params = { options: {} }
+  const settings = {}
+  const result = setOutputMode(params, settings)
+
+  expect(result).toEqual(SolidarityOutputMode.MODERATE)
+})
+
+test('settings output mode', () => {
+  const params = { options: {} }
+  const settings = { config: { output: 'SILENT' } }
+  const result = setOutputMode(params, settings)
+
+  expect(result).toEqual(SolidarityOutputMode.SILENT)
+})
+
+test('verbose output mode', () => {
+  const params = { options: { verbose: true } }
+  const settings = {}
+  const result = setOutputMode(params, settings)
+
+  expect(result).toEqual(SolidarityOutputMode.VERBOSE)
+})
+
+test('silent output mode', () => {
+  const params = { options: { silent: true } }
+  const settings = {}
+  const result = setOutputMode(params, settings)
+
+  expect(result).toEqual(SolidarityOutputMode.SILENT)
+})
+
+test('moderate output mode', () => {
+  const params = { options: { moderate: true } }
+  const settings = {}
+  const result = setOutputMode(params, settings)
+
+  expect(result).toEqual(SolidarityOutputMode.MODERATE)
+})
diff --git a/__tests__/commands/snapshot.ts b/__tests__/commands/snapshot.ts
index 6e6818e..b7a56be 100644
--- a/__tests__/commands/snapshot.ts
+++ b/__tests__/commands/snapshot.ts
@@ -45,18 +45,18 @@ describe('without a .solidarity file', () => {
     process.chdir(origCwd)
   })
 
-  // it('should prompt the user', async () => {
-  //   await snapshotCommand.run(context)
-  //   expect(context.prompt.ask.mock.calls).toEqual([
-  //     [
-  //       {
-  //         message: 'No `.solidarity` file found for this project.  Would you like to create one?',
-  //         name: 'createFile',
-  //         type: 'confirm',
-  //       },
-  //     ],
-  //   ])
-  // })
+  it('should prompt the user', async () => {
+    await snapshotCommand.run(context)
+    expect(context.prompt.ask.mock.calls).toEqual([
+      [
+        {
+          message: 'No `.solidarity` file found for this project.  Would you like to create one?',
+          name: 'createFile',
+          type: 'confirm',
+        },
+      ],
+    ])
+  })
 })
 
 describe('with a .solidarity file', () => {
diff --git a/__tests__/commands/solidarity.ts b/__tests__/commands/solidarity.ts
index eaebd8b..782466a 100644
--- a/__tests__/commands/solidarity.ts
+++ b/__tests__/commands/solidarity.ts
@@ -4,6 +4,7 @@ import { SolidarityOutputMode } from '../../src/types'
 const noConfigSolidarity = {
   checkRequirement: jest.fn(),
   getSolidaritySettings: jest.fn(() => ({})),
+  setOutputMode: require('../../src/extensions/functions/setOutputMode'), // don't mock setOutputMode
 }
 
 const verboseConfigSolidarity = {
diff --git a/__tests__/extensions/__snapshots__/extensionCheck.ts.snap b/__tests__/extensions/__snapshots__/extensionCheck.ts.snap
index d33967f..4dd5027 100644
--- a/__tests__/extensions/__snapshots__/extensionCheck.ts.snap
+++ b/__tests__/extensions/__snapshots__/extensionCheck.ts.snap
@@ -39,6 +39,7 @@ Object {
       "key": "command",
     },
   },
+  "setOutputMode": [Function],
   "setSolidaritySettings": [Function],
   "skipRule": [Function],
   "updateRequirement": [Function],