Repository: mochajs/mocha (js)

Issue:
🚀 Feature: Possibility to return 0 exit code when tests run successfully even with fails
**Is your feature request related to a problem or a nice-to-have?? Please describe.**
A clear and concise description of what the problem is. E.g. I'm always frustrated when [...]

Sometimes it is useful to separate situations when tests run successfully, and there is no infrastructure or runtime problems, and when there are some runtime problems.
Possibility to return non-zero exit code only on infrastructure or runtime problems make easier integration with different toolchain.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Flag, something like `failOnFailingTests` like in Karma (https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite).
- 0 exit code when tests run successfully
- 1 exit code when something went wrong

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Karma 
https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite

**Additional context**
Add any other context or screenshots about the feature request here.


Pull request:
feat: add option to not fail on failing test suite
### Requirements

* Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
* All new code requires tests to ensure against regressions.

### Description of the Change

Fixed #4216

Add possibility to distinguish 2 types of fails:
- Problem with test run
- Failing tests

When there are problems with test run, it is definitely problem.
When tests were run, but there are some test fails, it can be considered as ok.
It is useful to use Mocha in integrations with different build tools (for example [Gradle](https://gradle.org)), when we extract information from Mocha about tests and pass it to build tool and check exit code of Mocha.

Karma has [similar feature](https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#failonfailingtestsuite)
<!--

We must be able to understand the design of your change from this description. Keep in mind that the maintainers and/or community members reviewing this PR may not be familiar with the subsystem. Please be verbose.

-->

### Alternate Designs

It can be done with 2 runs of Mocha:
- with `dry-run` and check if JavaScript code can be run without problems
- as usual - to run tests

But it requires 2 runs, which is not very convenient.

### Why should this be in core?

When using Mocha and look at exit code, it is impossible to distinguish 2 cases
- 1 failing test (exit code = 1)
- Fail with running test (exit code = 1)

But these 2 cases can be completely different
<!-- Explain why this functionality should be in mocha as opposed to its own package -->

### Benefits
New useful feature for build pipelines integrations

### Possible Drawbacks

<!-- What are the possible side-effects or negative impacts of the code change? -->

### Applicable issues

#4216 

<!--
* Enter any applicable Issues here.

* Mocha follows semantic versioning: http://semver.org

* Is this a breaking change (major release)?
* Is it an enhancement (minor release)?
* Is it a bug fix, or does it not impact production code (patch release)?
-->


Patch:
diff --git a/docs/index.md b/docs/index.md
index cc3d9eb2..cbd22d2d 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -915,6 +915,10 @@ To ensure your tests aren't leaving messes around, here are some ideas to get st
 - Try something like [wtfnode][npm-wtfnode]
 - Use [`.only`](#exclusive-tests) until you find the test that causes Mocha to hang
 
+### `--pass-on-failing-test-suite`
+
+If set to `true`, Mocha returns exit code `0` even if there are failed tests during run.
+
 ### `--fail-zero`
 
 > _New in v9.1.0_ Fail test run if no tests are encountered with `exit-code: 1`.
diff --git a/lib/cli/run-helpers.js b/lib/cli/run-helpers.js
index 0d01afbf..ec65829f 100644
--- a/lib/cli/run-helpers.js
+++ b/lib/cli/run-helpers.js
@@ -21,25 +21,24 @@ const {UnmatchedFile} = require('./collect-files');
 
 /**
  * Exits Mocha when tests + code under test has finished execution (default)
- * @param {number} code - Exit code; typically # of failures
+ * @param {number} clampedCode - Exit code; typically # of failures
  * @ignore
  * @private
  */
-const exitMochaLater = code => {
+const exitMochaLater = clampedCode => {
   process.on('exit', () => {
-    process.exitCode = Math.min(code, 255);
+    process.exitCode = clampedCode;
   });
 };
 
 /**
  * Exits Mocha when Mocha itself has finished execution, regardless of
  * what the tests or code under test is doing.
- * @param {number} code - Exit code; typically # of failures
+ * @param {number} clampedCode - Exit code; typically # of failures
  * @ignore
  * @private
  */
-const exitMocha = code => {
-  const clampedCode = Math.min(code, 255);
+const exitMocha = clampedCode => {
   let draining = 0;
 
   // Eagerly set the process's exit code in case stream.write doesn't
@@ -139,12 +138,17 @@ const handleUnmatchedFiles = (mocha, unmatchedFiles) => {
  * @param {Mocha} mocha - Mocha instance
  * @param {Options} [opts] - Command line options
  * @param {boolean} [opts.exit] - Whether or not to force-exit after tests are complete
+ * @param {boolean} [opts.passOnFailingTestSuite] - Whether or not to fail test run if tests were failed
  * @param {Object} fileCollectParams - Parameters that control test
  *   file collection. See `lib/cli/collect-files.js`.
  * @returns {Promise<Runner>}
  * @private
  */
-const singleRun = async (mocha, {exit}, fileCollectParams) => {
+const singleRun = async (
+  mocha,
+  {exit, passOnFailingTestSuite},
+  fileCollectParams
+) => {
   const fileCollectionObj = collectFiles(fileCollectParams);
 
   if (fileCollectionObj.unmatchedFiles.length > 0) {
@@ -156,7 +160,9 @@ const singleRun = async (mocha, {exit}, fileCollectParams) => {
 
   // handles ESM modules
   await mocha.loadFilesAsync();
-  return mocha.run(exit ? exitMocha : exitMochaLater);
+  return mocha.run(
+    createExitHandler({exit, passOnFailingTestSuite})
+  );
 };
 
 /**
@@ -186,7 +192,9 @@ const parallelRun = async (mocha, options, fileCollectParams) => {
   mocha.files = fileCollectionObj.files;
 
   // note that we DO NOT load any files here; this is handled by the worker
-  return mocha.run(options.exit ? exitMocha : exitMochaLater);
+  return mocha.run(
+    createExitHandler(options)
+  );
 };
 
 /**
@@ -282,3 +290,15 @@ exports.validateLegacyPlugin = (opts, pluginType, map = {}) => {
     }
   }
 };
+
+const createExitHandler = ({ exit, passOnFailingTestSuite }) => {
+  return code => {
+    const clampedCode = passOnFailingTestSuite
+      ? 0
+      : Math.min(code, 255);
+
+    return exit
+      ? exitMocha(clampedCode)
+      : exitMochaLater(clampedCode);
+  };
+};
diff --git a/lib/cli/run-option-metadata.js b/lib/cli/run-option-metadata.js
index 492608fb..83aa70dd 100644
--- a/lib/cli/run-option-metadata.js
+++ b/lib/cli/run-option-metadata.js
@@ -35,6 +35,7 @@ const TYPES = (exports.types = {
     'diff',
     'dry-run',
     'exit',
+    'pass-on-failing-test-suite',
     'fail-zero',
     'forbid-only',
     'forbid-pending',
diff --git a/lib/cli/run.js b/lib/cli/run.js
index 66c8cbbb..3389e6df 100644
--- a/lib/cli/run.js
+++ b/lib/cli/run.js
@@ -98,6 +98,11 @@ exports.builder = yargs =>
         requiresArg: true,
         coerce: list
       },
+      'pass-on-failing-test-suite': {
+        default: false,
+        description: 'Not fail test run if tests were failed',
+        group: GROUPS.RULES
+      },
       'fail-zero': {
         description: 'Fail test run if no test(s) encountered',
         group: GROUPS.RULES
diff --git a/lib/mocha.js b/lib/mocha.js
index f93865df..c6ee2485 100644
--- a/lib/mocha.js
+++ b/lib/mocha.js
@@ -157,6 +157,7 @@ exports.run = function (...args) {
  * @param {boolean} [options.delay] - Delay root suite execution?
  * @param {boolean} [options.diff] - Show diff on failure?
  * @param {boolean} [options.dryRun] - Report tests without running them?
+ * @param {boolean} [options.passOnFailingTestSuite] - Fail test run if tests were failed?
  * @param {boolean} [options.failZero] - Fail test run if zero tests?
  * @param {string} [options.fgrep] - Test filter given string.
  * @param {boolean} [options.forbidOnly] - Tests marked `only` fail the suite?
@@ -216,6 +217,7 @@ function Mocha(options = {}) {
     'delay',
     'diff',
     'dryRun',
+    'passOnFailingTestSuite',
     'failZero',
     'forbidOnly',
     'forbidPending',
@@ -870,6 +872,20 @@ Mocha.prototype.failZero = function (failZero) {
   return this;
 };
 
+/**
+ * Fail test run if tests were failed.
+ *
+ * @public
+ * @see [CLI option](../#-pass-on-failing-test-suite)
+ * @param {boolean} [passOnFailingTestSuite=false] - Whether to fail test run.
+ * @return {Mocha} this
+ * @chainable
+ */
+Mocha.prototype.passOnFailingTestSuite = function(passOnFailingTestSuite) {
+  this.options.passOnFailingTestSuite = passOnFailingTestSuite === true;
+  return this;
+};
+
 /**
  * Causes tests marked `only` to fail the suite.
  *


Test patch:
diff --git a/test/integration/fixtures/failing-sync.fixture.js b/test/integration/fixtures/failing-sync.fixture.js
new file mode 100644
index 00000000..f81f9653
--- /dev/null
+++ b/test/integration/fixtures/failing-sync.fixture.js
@@ -0,0 +1,9 @@
+'use strict';
+
+var assert = require('assert');
+
+describe('a suite', function() {
+  it('should succeed', function() {
+    assert(false);
+  });
+});
diff --git a/test/integration/options/passOnFailingTestSuite.spec.js b/test/integration/options/passOnFailingTestSuite.spec.js
new file mode 100644
index 00000000..9f902da7
--- /dev/null
+++ b/test/integration/options/passOnFailingTestSuite.spec.js
@@ -0,0 +1,40 @@
+'use strict';
+
+var helpers = require('../helpers');
+var runMochaJSON = helpers.runMochaJSON;
+
+describe('Enabled --pass-on-failing-test-suite', function() {
+  var args = ['--pass-on-failing-test-suite=true'];
+
+  it('Test should finish with zero code with disabled option', function(done) {
+    var fixture = 'failing-sync.fixture.js';
+    runMochaJSON(fixture, args, function(err, res) {
+      if (err) {
+        return done(err);
+      }
+
+      expect(res, 'to have passed test count', 0)
+        .and('to have test count', 1)
+        .and('to have exit code', 0);
+      done();
+    });
+  });
+});
+
+describe('Disabled --pass-on-failing-test-suite', function() {
+  var args = ['--pass-on-failing-test-suite=false'];
+
+  it('Test should return non-zero code with enabled option', function(done) {
+    var fixture = 'failing-sync.fixture.js';
+    runMochaJSON(fixture, args, function(err, res) {
+      if (err) {
+        return done(err);
+      }
+
+      expect(res, 'to have passed test count', 0)
+        .and('to have test count', 1)
+        .and('to have exit code', 1);
+      done();
+    });
+  });
+});
diff --git a/test/unit/mocha.spec.js b/test/unit/mocha.spec.js
index f966e0c3..e0f60357 100644
--- a/test/unit/mocha.spec.js
+++ b/test/unit/mocha.spec.js
@@ -376,6 +376,28 @@ describe('Mocha', function () {
       });
     });
 
+    describe('passOnFailingTestSuite()', function() {
+      it('should set the passOnFailingTestSuite option to false', function() {
+        mocha.passOnFailingTestSuite();
+        expect(
+          mocha.options,
+          'to have property',
+          'passOnFailingTestSuite',
+          false
+        );
+      });
+
+      it('should set the passOnFailingTestSuite option to true', function() {
+        mocha.passOnFailingTestSuite(true);
+        expect(
+          mocha.options,
+          'to have property',
+          'passOnFailingTestSuite',
+          true
+        );
+      });
+    });
+
     describe('failZero()', function () {
       it('should set the failZero option to true', function () {
         mocha.failZero();