Repository: firebase/firebase-tools (ts)

Issue:
Request: Automatically retry failed function deploys that fail due to "You have exceeded your deployment quota"
## Problem & Background

We have a deployment of ~70 functions and deploy them as part of CI at once. Whenever we deploy a random set of functions fails to deploy due to "You have exceeded your deployment quota". This used to block us completely a few weeks ago when there still was a quota on build time. Since this has been moved to Cloud Build as far as I can tell there are no more limits and you simply pay for your build time. Which is awesome, no more blocking the build due to quotas!

Unfortunately there still seems to be some other quotas/rate limits that are causing the deploy to fail. From what I can tell these might be simply too many write requests against the Cloud Functions API (or perhaps cloud build?).   

The error message suggests to deploy with --only - Unfortunately it is not easy for us to split these up into separately deployed functions. It is also impossible to do when a dependency is updated or we change a data model or utility library that is used by different functions. Analyzing which function changed and has to be redeployed is not possible (for us) automatically and has to be done manually. This then brings new pains, where automated deploys after code review become impossible.

Right now this means our pipelines all fail regularily - and we manually retry them until every functions deployed successfully once. 
![image](https://user-images.githubusercontent.com/337968/92576631-087c6480-f28a-11ea-9692-a198a8866bd3.png)

## Suggestion

Since retrying the failed functions right after always works, my suggestion would be to retry failed deploys that hit code 8 here:

https://github.com/firebase/firebase-tools/blob/3994bf7209a71ed5dde100c69ca96dc2d9c3634e/src/deploy/functions/release.js#L97-L105

I suppose this could be implemented here:

https://github.com/firebase/firebase-tools/blob/3994bf7209a71ed5dde100c69ca96dc2d9c3634e/src/deploy/functions/release.js#L526

If this would be a way forward I'm happy to create a PR for this. 


Pull request description:
Retry 'Quota Exceeded' errors during function deployment
### Description
Adds logic to retry 'Quota Exceeded' errors during functions deployment. Fixes #2606 and #1372.

I also took this opportunity to deduplicate + test some error handling logic that was repeated across the different tasks by moving it into handler functions for the throttler.

### Scenarios Tested
I tested this by deploying 240 functions across 3 regions, and was able to do so successfully (with quite a few quota error retries).
I also tested out `functions:delete` by successfully deleting a large group of functions.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dfb54c6a9..c2061be53 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
-- Fixes native module issues by removing `fast-crc32c` (#3247, #3239)
+- Fixes native module issues by removing `fast-crc32c` (#3247, #3239).
+- Adds retries on Quota Exceeded errors during functions deployment (#2606, #1372).
 - Update to Cloud Firestore emulator v1.11.13 which includes a bug fix for documents which are created and deleted in a single transaction.
 - Fixes an edge case with nextPageToken in batchGet in Auth Emulator (#3231).
 - Removes unused dependencies (#3252).
diff --git a/src/deploy/functions/errorHandler.ts b/src/deploy/functions/errorHandler.ts
index 52848e1f7..c2b5875bf 100644
--- a/src/deploy/functions/errorHandler.ts
+++ b/src/deploy/functions/errorHandler.ts
@@ -3,14 +3,7 @@ import * as clc from "cli-color";
 import { logger } from "../../logger";
 import { getFunctionId, getFunctionLabel } from "../../functionsDeployHelper";
 import { FirebaseError } from "../../error";
-
-type OperationType =
-  | "create"
-  | "update"
-  | "delete"
-  | "upsert schedule"
-  | "delete schedule"
-  | "make public";
+import { OperationType } from "./tasks";
 
 type Level = "error" | "warning";
 
diff --git a/src/deploy/functions/release.ts b/src/deploy/functions/release.ts
index dffa42219..643569a94 100644
--- a/src/deploy/functions/release.ts
+++ b/src/deploy/functions/release.ts
@@ -5,7 +5,7 @@ import * as clc from "cli-color";
 
 import * as utils from "../../utils";
 import * as helper from "../../functionsDeployHelper";
-import { CloudFunctionTrigger, createDeploymentPlan } from "./deploymentPlanner";
+import { createDeploymentPlan } from "./deploymentPlanner";
 import * as tasks from "./tasks";
 import { getAppEngineLocation } from "../../functionsConfig";
 import { promptForFunctionDeletion } from "./prompts";
@@ -30,15 +30,26 @@ export async function release(context: any, options: any, payload: any) {
     context.existingFunctions,
     context.filters
   );
-  const cloudFunctionsQueue = new Queue<() => Promise<CloudFunctionTrigger | void>, void>({});
-  const schedulerQueue = new Queue<() => Promise<any>, void>({});
+
+  // This queue needs to retry quota errors.
+  // The main quotas that can be exceeded are per 1 minute quotas,
+  // so we start with a larger backoff to reduce the liklihood of extra retries.
+  const cloudFunctionsQueue = new Queue<tasks.DeploymentTask, void>({
+    retries: 30,
+    backoff: 20000,
+    concurrency: 40,
+    maxBackoff: 40000,
+    handler: tasks.functionsDeploymentHandler(timer, errorHandler),
+  });
+  const schedulerQueue = new Queue<tasks.DeploymentTask, void>({
+    handler: tasks.schedulerDeploymentHandler(errorHandler),
+  });
   const regionPromises = [];
 
   const taskParams: tasks.TaskParams = {
     projectId,
     sourceUrl,
     runtime: context.runtimeChoice,
-    timer,
     errorHandler,
   };
 
@@ -94,7 +105,20 @@ export async function release(context: any, options: any, payload: any) {
   // Wait for the first function in each region to be deployed, and all the other calls to be queued,
   // then close the queue.
   // Wait for all of the deployments to complete.
-  await Promise.all(queuePromises);
+  try {
+    await Promise.all(queuePromises);
+  } catch (err) {
+    utils.reject(
+      "Exceeded maximum retries while deploying functions. " +
+        "If you are deploying a large number of functions, " +
+        "please deploy your functions in batches by using the --only flag, " +
+        "and wait a few minutes before deploying again. " +
+        "Go to https://firebase.google.com/docs/cli/#partial_deploys to learn more.",
+      {
+        original: err,
+      }
+    );
+  }
   helper.logAndTrackDeployStats(cloudFunctionsQueue, errorHandler);
   helper.printTriggerUrls(projectId, sourceUrl);
   errorHandler.printWarnings();
diff --git a/src/deploy/functions/tasks.ts b/src/deploy/functions/tasks.ts
index 3ab3bfb71..e720e6f3f 100644
--- a/src/deploy/functions/tasks.ts
+++ b/src/deploy/functions/tasks.ts
@@ -14,6 +14,7 @@ import { getHumanFriendlyRuntimeName } from "../../parseRuntimeAndValidateSDK";
 import { deleteTopic } from "../../gcp/pubsub";
 import { DeploymentTimer } from "./deploymentTimer";
 import { ErrorHandler } from "./errorHandler";
+import { result } from "lodash";
 
 // TODO: Tune this for better performance.
 const defaultPollerOptions = {
@@ -22,21 +23,38 @@ const defaultPollerOptions = {
   masterTimeout: 25 * 60000, // 25 minutes is the maximum build time for a function
 };
 
+export type OperationType =
+  | "create"
+  | "update"
+  | "delete"
+  | "upsert schedule"
+  | "delete schedule"
+  | "make public";
+
+export interface DeploymentTask {
+  run: () => Promise<any>;
+  functionName: string;
+  operationType: OperationType;
+}
+
 export interface TaskParams {
   projectId: string;
   runtime?: string;
   sourceUrl?: string;
-  sourceToken?: string;
-  timer: DeploymentTimer;
   errorHandler: ErrorHandler;
 }
 
+/**
+ * Cloud Functions Deployments Tasks and Handler
+ */
+
 export function createFunctionTask(
   params: TaskParams,
   fn: CloudFunctionTrigger,
+  sourceToken?: string,
   onPoll?: (op: OperationResult<CloudFunctionTrigger>) => void
-): () => Promise<CloudFunctionTrigger | void> {
-  return async () => {
+): DeploymentTask {
+  const run = async () => {
     utils.logBullet(
       clc.bold.cyan("functions: ") +
         "creating " +
@@ -45,63 +63,62 @@ export function createFunctionTask(
         clc.bold(helper.getFunctionLabel(fn.name)) +
         "..."
     );
-    params.timer.startTimer(fn.name, "create");
     const eventType = fn.eventTrigger ? fn.eventTrigger.eventType : "https";
-    try {
-      const createRes = await cloudfunctions.createFunction({
-        projectId: params.projectId,
-        region: helper.getRegion(fn.name),
-        eventType: eventType,
-        functionName: helper.getFunctionId(fn.name),
-        entryPoint: fn.entryPoint,
-        trigger: helper.getFunctionTrigger(fn),
-        labels: Object.assign({}, deploymentTool.labels(), fn.labels),
-        sourceUploadUrl: params.sourceUrl,
-        sourceToken: params.sourceToken,
-        runtime: params.runtime,
-        availableMemoryMb: fn.availableMemoryMb,
-        timeout: fn.timeout,
-        maxInstances: fn.maxInstances,
-        environmentVariables: fn.environmentVariables,
-        vpcConnector: fn.vpcConnector,
-        vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
-        serviceAccountEmail: fn.serviceAccountEmail,
-        ingressSettings: fn.ingressSettings,
-      });
-      const pollerOptions: OperationPollerOptions = Object.assign(
-        {
-          pollerName: `create-${fn.name}`,
-          operationResourceName: createRes.name,
-          onPoll,
-        },
-        defaultPollerOptions
-      );
-      const operationResult = await pollOperation<CloudFunctionTrigger>(pollerOptions);
-      if (eventType === "https") {
-        try {
-          await cloudfunctions.setIamPolicy({
-            name: fn.name,
-            policy: cloudfunctions.DEFAULT_PUBLIC_POLICY,
-          });
-        } catch (err) {
-          params.errorHandler.record("warning", fn.name, "make public", err.message);
-        }
+    const createRes = await cloudfunctions.createFunction({
+      projectId: params.projectId,
+      region: helper.getRegion(fn.name),
+      eventType: eventType,
+      functionName: helper.getFunctionId(fn.name),
+      entryPoint: fn.entryPoint,
+      trigger: helper.getFunctionTrigger(fn),
+      labels: Object.assign({}, deploymentTool.labels(), fn.labels),
+      sourceUploadUrl: params.sourceUrl,
+      sourceToken: sourceToken,
+      runtime: params.runtime,
+      availableMemoryMb: fn.availableMemoryMb,
+      timeout: fn.timeout,
+      maxInstances: fn.maxInstances,
+      environmentVariables: fn.environmentVariables,
+      vpcConnector: fn.vpcConnector,
+      vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
+      serviceAccountEmail: fn.serviceAccountEmail,
+      ingressSettings: fn.ingressSettings,
+    });
+    const pollerOptions: OperationPollerOptions = Object.assign(
+      {
+        pollerName: `create-${fn.name}`,
+        operationResourceName: createRes.name,
+        onPoll,
+      },
+      defaultPollerOptions
+    );
+    const operationResult = await pollOperation<CloudFunctionTrigger>(pollerOptions);
+    if (eventType === "https") {
+      try {
+        await cloudfunctions.setIamPolicy({
+          name: fn.name,
+          policy: cloudfunctions.DEFAULT_PUBLIC_POLICY,
+        });
+      } catch (err) {
+        params.errorHandler.record("warning", fn.name, "make public", err.original.message);
       }
-      params.timer.endTimer(fn.name);
-      helper.printSuccess(fn.name, "create");
-      return operationResult;
-    } catch (err) {
-      params.errorHandler.record("error", fn.name, "create", err.message || "");
     }
+    return operationResult;
+  };
+  return {
+    run,
+    functionName: fn.name,
+    operationType: "create",
   };
 }
 
 export function updateFunctionTask(
   params: TaskParams,
   fn: CloudFunctionTrigger,
+  sourceToken?: string,
   onPoll?: (op: OperationResult<CloudFunctionTrigger>) => void
-): () => Promise<CloudFunctionTrigger | void> {
-  return async () => {
+): DeploymentTask {
+  const run = async () => {
     utils.logBullet(
       clc.bold.cyan("functions: ") +
         "updating " +
@@ -110,122 +127,96 @@ export function updateFunctionTask(
         clc.bold(helper.getFunctionLabel(fn.name)) +
         "..."
     );
-    params.timer.startTimer(fn.name, "update");
     const eventType = fn.eventTrigger ? fn.eventTrigger.eventType : "https";
-    try {
-      const updateRes = await cloudfunctions.updateFunction({
-        projectId: params.projectId,
-        region: helper.getRegion(fn.name),
-        eventType: eventType,
-        functionName: helper.getFunctionId(fn.name),
-        entryPoint: fn.entryPoint,
-        trigger: helper.getFunctionTrigger(fn),
-        labels: Object.assign({}, deploymentTool.labels(), fn.labels),
-        sourceUploadUrl: params.sourceUrl,
-        sourceToken: params.sourceToken,
-        runtime: params.runtime,
-        availableMemoryMb: fn.availableMemoryMb,
-        timeout: fn.timeout,
-        maxInstances: fn.maxInstances,
-        environmentVariables: fn.environmentVariables,
-        vpcConnector: fn.vpcConnector,
-        vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
-        serviceAccountEmail: fn.serviceAccountEmail,
-        ingressSettings: fn.ingressSettings,
-      });
-      const pollerOptions: OperationPollerOptions = Object.assign(
-        {
-          pollerName: `update-${fn.name}`,
-          operationResourceName: updateRes.name,
-          onPoll,
-        },
-        defaultPollerOptions
-      );
-      const operationResult = await pollOperation<CloudFunctionTrigger>(pollerOptions);
-      params.timer.endTimer(fn.name);
-      helper.printSuccess(fn.name, "update");
-      return operationResult;
-    } catch (err) {
-      params.errorHandler.record("error", fn.name, "update", err.message || "");
-    }
+    const updateRes = await cloudfunctions.updateFunction({
+      projectId: params.projectId,
+      region: helper.getRegion(fn.name),
+      eventType: eventType,
+      functionName: helper.getFunctionId(fn.name),
+      entryPoint: fn.entryPoint,
+      trigger: helper.getFunctionTrigger(fn),
+      labels: Object.assign({}, deploymentTool.labels(), fn.labels),
+      sourceUploadUrl: params.sourceUrl,
+      sourceToken: sourceToken,
+      runtime: params.runtime,
+      availableMemoryMb: fn.availableMemoryMb,
+      timeout: fn.timeout,
+      maxInstances: fn.maxInstances,
+      environmentVariables: fn.environmentVariables,
+      vpcConnector: fn.vpcConnector,
+      vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
+      serviceAccountEmail: fn.serviceAccountEmail,
+      ingressSettings: fn.ingressSettings,
+    });
+    const pollerOptions: OperationPollerOptions = Object.assign(
+      {
+        pollerName: `update-${fn.name}`,
+        operationResourceName: updateRes.name,
+        onPoll,
+      },
+      defaultPollerOptions
+    );
+    const operationResult = await pollOperation<CloudFunctionTrigger>(pollerOptions);
+    return operationResult;
+  };
+  return {
+    run,
+    functionName: fn.name,
+    operationType: "update",
   };
 }
 
-export function deleteFunctionTask(params: TaskParams, fnName: string) {
-  return async () => {
+export function deleteFunctionTask(params: TaskParams, fnName: string): DeploymentTask {
+  const run = async () => {
     utils.logBullet(
       clc.bold.cyan("functions: ") +
         "deleting function " +
         clc.bold(helper.getFunctionLabel(fnName)) +
         "..."
     );
-    params.timer.startTimer(fnName, "delete");
-    try {
-      const deleteRes = await cloudfunctions.deleteFunction({
-        functionName: fnName,
-      });
-      const pollerOptions: OperationPollerOptions = Object.assign(
-        {
-          pollerName: `delete-${fnName}`,
-          operationResourceName: deleteRes.name,
-        },
-        defaultPollerOptions
-      );
-      const operationResult = await pollOperation<void>(pollerOptions);
-      params.timer.endTimer(fnName);
-      helper.printSuccess(fnName, "delete");
-      return operationResult;
-    } catch (err) {
-      params.errorHandler.record("error", fnName, "delete", err.message || "");
-    }
+    const deleteRes = await cloudfunctions.deleteFunction({
+      functionName: fnName,
+    });
+    const pollerOptions: OperationPollerOptions = Object.assign(
+      {
+        pollerName: `delete-${fnName}`,
+        operationResourceName: deleteRes.name,
+      },
+      defaultPollerOptions
+    );
+    return await pollOperation<void>(pollerOptions);
   };
-}
-
-export function upsertScheduleTask(
-  params: TaskParams,
-  fn: CloudFunctionTrigger,
-  appEngineLocation: string
-): () => Promise<any> {
-  return async () => {
-    const job = helper.toJob(fn, appEngineLocation, params.projectId);
-    try {
-      await cloudscheduler.createOrReplaceJob(job);
-      helper.printSuccess(fn.name, "upsert schedule");
-    } catch (err) {
-      params.errorHandler.record("error", fn.name, "upsert schedule", err.message || "");
-    }
+  return {
+    run,
+    functionName: fnName,
+    operationType: "delete",
   };
 }
 
-export function deleteScheduleTask(
-  params: TaskParams,
-  fnName: string,
-  appEngineLocation: string
-): () => Promise<void> {
-  return async () => {
-    const jobName = helper.getScheduleName(fnName, appEngineLocation);
-    const topicName = helper.getTopicName(fnName);
+export function functionsDeploymentHandler(
+  timer: DeploymentTimer,
+  errorHandler: ErrorHandler
+): (task: DeploymentTask) => Promise<any | undefined> {
+  return async (task: DeploymentTask) => {
+    let result;
     try {
-      await cloudscheduler.deleteJob(jobName);
+      timer.startTimer(task.functionName, task.operationType);
+      result = await task.run();
+      helper.printSuccess(task.functionName, task.operationType);
     } catch (err) {
-      // If the job has already been deleted, don't throw an error.
-      if (err.status !== 404) {
-        params.errorHandler.record("error", fnName, "delete schedule", err.message || "");
-        return;
+      if (err.original?.context?.response?.statusCode === 429) {
+        // Throw quota errors so that throttler retries them.
+        throw err;
       }
-      logger.debug(`Scheduler job ${jobName} not found.`);
-    }
-    try {
-      await deleteTopic(topicName);
-      helper.printSuccess(fnName, "delete schedule");
-    } catch (err) {
-      // If the topic has already been deleted, don't throw an error.
-      if (err.status !== 404) {
-        params.errorHandler.record("error", fnName, "delete schedule", err.message || "");
-        return;
-      }
-      logger.debug(`Scheduler topic ${topicName} not found.`);
+      errorHandler.record(
+        "error",
+        task.functionName,
+        task.operationType,
+        err.original?.message || ""
+      );
     }
+    timer.endTimer(task.functionName);
+    return result;
   };
 }
 
@@ -235,7 +226,7 @@ export function deleteScheduleTask(
 export function runRegionalFunctionDeployment(
   params: TaskParams,
   regionalDeployment: RegionalDeployment,
-  queue: Queue<() => Promise<any>, void>
+  queue: Queue<DeploymentTask, void>
 ): Promise<void> {
   // Build an onPoll function to check for sourceToken and queue up the rest of the deployment.
   const onPollFn = (op: any) => {
@@ -256,11 +247,11 @@ export function runRegionalFunctionDeployment(
   // Choose a first function to deploy.
   if (regionalDeployment.functionsToCreate.length) {
     const firstFn = regionalDeployment.functionsToCreate.shift()!;
-    const task = createFunctionTask(params, firstFn!, onPollFn);
+    const task = createFunctionTask(params, firstFn!, /* sourceToken= */ undefined, onPollFn);
     return queue.run(task);
   } else if (regionalDeployment.functionsToUpdate.length) {
     const firstFn = regionalDeployment.functionsToUpdate.shift()!;
-    const task = updateFunctionTask(params, firstFn!, onPollFn);
+    const task = updateFunctionTask(params, firstFn!, /* sourceToken= */ undefined, onPollFn);
     return queue.run(task);
   }
   // If there are no functions to create or update in this region, no need to do anything.
@@ -270,13 +261,71 @@ export function runRegionalFunctionDeployment(
 function finishRegionalFunctionDeployment(
   params: TaskParams,
   regionalDeployment: RegionalDeployment,
-  queue: Queue<() => Promise<any>, void>
+  queue: Queue<DeploymentTask, void>
 ): void {
-  params.sourceToken = regionalDeployment.sourceToken;
   for (const fn of regionalDeployment.functionsToCreate) {
-    queue.run(createFunctionTask(params, fn));
+    queue.run(createFunctionTask(params, fn, regionalDeployment.sourceToken));
   }
   for (const fn of regionalDeployment.functionsToUpdate) {
-    queue.run(updateFunctionTask(params, fn));
+    queue.run(updateFunctionTask(params, fn, regionalDeployment.sourceToken));
   }
 }
+
+/**
+ * Cloud Scheduler Deployments Tasks and Handler
+ */
+
+export function upsertScheduleTask(
+  params: TaskParams,
+  fn: CloudFunctionTrigger,
+  appEngineLocation: string
+): DeploymentTask {
+  const run = async () => {
+    const job = helper.toJob(fn, appEngineLocation, params.projectId);
+    return await cloudscheduler.createOrReplaceJob(job);
+  };
+  return {
+    run,
+    functionName: fn.name,
+    operationType: "upsert schedule",
+  };
+}
+
+export function deleteScheduleTask(
+  params: TaskParams,
+  fnName: string,
+  appEngineLocation: string
+): DeploymentTask {
+  const run = async () => {
+    const jobName = helper.getScheduleName(fnName, appEngineLocation);
+    const topicName = helper.getTopicName(fnName);
+    await cloudscheduler.deleteJob(jobName);
+    await deleteTopic(topicName);
+  };
+  return {
+    run,
+    functionName: fnName,
+    operationType: "delete schedule",
+  };
+}
+
+export function schedulerDeploymentHandler(
+  errorHandler: ErrorHandler
+): (task: DeploymentTask) => Promise<any | undefined> {
+  return async (task: DeploymentTask) => {
+    let result;
+    try {
+      result = await task.run();
+      helper.printSuccess(task.functionName, task.operationType);
+    } catch (err) {
+      if (err.status === 429) {
+        // Throw quota errors so that throttler retries them.
+        throw err;
+      } else if (err.status !== 404) {
+        // Ignore 404 errors from scheduler calls since they may be deleted out of band.
+        errorHandler.record("error", task.functionName, task.operationType, err.message || "");
+      }
+    }
+    return result;
+  };
+}
diff --git a/src/functionsDelete.ts b/src/functionsDelete.ts
index fb228185f..141ba7449 100644
--- a/src/functionsDelete.ts
+++ b/src/functionsDelete.ts
@@ -10,13 +10,21 @@ export async function deleteFunctions(
   projectId: string,
   appEngineLocation: string
 ): Promise<void> {
-  const cloudFunctionsQueue = new Queue<() => Promise<void>, void>({});
-  const schedulerQueue = new Queue<() => Promise<void>, void>({});
   const timer = new DeploymentTimer();
   const errorHandler = new ErrorHandler();
+  const cloudFunctionsQueue = new Queue<tasks.DeploymentTask, void>({
+    handler: tasks.functionsDeploymentHandler(timer, errorHandler),
+    retries: 30,
+    backoff: 10000,
+    concurrency: 40,
+    maxBackoff: 40000,
+  });
+  const schedulerQueue = new Queue<tasks.DeploymentTask, void>({
+    handler: tasks.schedulerDeploymentHandler(errorHandler),
+  });
+
   const taskParams = {
     projectId,
-    timer,
     errorHandler,
   };
   functionsNamesToDelete.forEach((fnName) => {
diff --git a/src/gcp/cloudfunctions.ts b/src/gcp/cloudfunctions.ts
index a560c6684..e8ef1a6d1 100644
--- a/src/gcp/cloudfunctions.ts
+++ b/src/gcp/cloudfunctions.ts
@@ -39,15 +39,16 @@ export const DEFAULT_PUBLIC_POLICY = {
  * @param err The error returned from the operation.
  */
 function functionsOpLogReject(funcName: string, type: string, err: any): void {
-  utils.logWarning(clc.bold.yellow("functions:") + " failed to " + type + " function " + funcName);
   if (err?.context?.response?.statusCode === 429) {
-    logger.debug(err.message);
-    logger.info(
-      "You have exceeded your deployment quota, please deploy your functions in batches by using the --only flag, " +
-        "and wait a few minutes before deploying again. Go to https://firebase.google.com/docs/cli/#partial_deploys to learn more."
+    utils.logWarning(
+      `${clc.bold.yellow(
+        "functions:"
+      )} got "Quota Exceeded" error while trying to ${type} ${funcName}. Waiting to retry...`
     );
   } else {
-    logger.info(err.message);
+    utils.logWarning(
+      clc.bold.yellow("functions:") + " failed to " + type + " function " + funcName
+    );
   }
   throw new FirebaseError(`Failed to ${type} function ${funcName}`, {
     original: err,


Test patch:
diff --git a/src/test/deploy/functions/tasks.spec.ts b/src/test/deploy/functions/tasks.spec.ts
new file mode 100644
index 000000000..8db0941ef
--- /dev/null
+++ b/src/test/deploy/functions/tasks.spec.ts
@@ -0,0 +1,194 @@
+import { expect } from "chai";
+import * as sinon from "sinon";
+
+import * as tasks from "../../../deploy/functions/tasks";
+import { DeploymentTimer } from "../../../deploy/functions/deploymentTimer";
+import { ErrorHandler } from "../../../deploy/functions/errorHandler";
+import { FirebaseError } from "../../../error";
+
+describe("Function Deployment tasks", () => {
+  describe("functionsDeploymentHandler", () => {
+    let sandbox: sinon.SinonSandbox;
+    let timerStub: sinon.SinonStubbedInstance<DeploymentTimer>;
+    let errorHandlerStub: sinon.SinonStubbedInstance<ErrorHandler>;
+
+    beforeEach(() => {
+      sandbox = sinon.createSandbox();
+      timerStub = sandbox.createStubInstance(DeploymentTimer);
+      errorHandlerStub = sandbox.createStubInstance(ErrorHandler);
+    });
+
+    afterEach(() => {
+      sandbox.restore();
+    });
+
+    it("should execute the task and time it", async () => {
+      const run = sinon.spy();
+      const functionName = "myFunc";
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: functionName,
+        operationType: "create",
+      };
+
+      const handler = tasks.functionsDeploymentHandler(timerStub, errorHandlerStub);
+      await handler(testTask);
+
+      expect(timerStub.startTimer).to.have.been.calledWith(functionName);
+      expect(run).to.have.been.called;
+      expect(timerStub.endTimer).to.have.been.calledWith(functionName);
+      expect(errorHandlerStub.record).not.to.have.been.called;
+    });
+
+    it("should throw quota errors", async () => {
+      const originalError = {
+        name: "Quota Exceeded",
+        message: "an error occurred",
+        context: {
+          response: {
+            statusCode: 429,
+          },
+        },
+      };
+      const run = sinon.spy(() => {
+        throw new FirebaseError("an error occurred", {
+          original: originalError,
+        });
+      });
+      const functionName = "myFunc";
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: functionName,
+        operationType: "create",
+      };
+
+      const handler = tasks.functionsDeploymentHandler(timerStub, errorHandlerStub);
+
+      await expect(handler(testTask)).to.eventually.be.rejected;
+
+      expect(run).to.have.been.called;
+      expect(errorHandlerStub.record).not.to.have.been.called;
+    });
+
+    it("should handle other errors", async () => {
+      const originalError = {
+        name: "Some Other Error",
+        message: "an error occurred",
+        context: {
+          response: {
+            statusCode: 500,
+          },
+        },
+      };
+      const run = sinon.spy(() => {
+        throw new FirebaseError("an error occurred", {
+          original: originalError,
+        });
+      });
+      const functionName = "myFunc";
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: functionName,
+        operationType: "create",
+      };
+
+      const handler = tasks.functionsDeploymentHandler(timerStub, errorHandlerStub);
+      await handler(testTask);
+
+      expect(timerStub.startTimer).to.have.been.calledWith(functionName);
+      expect(run).to.have.been.called;
+      expect(timerStub.endTimer).to.have.been.calledWith(functionName);
+      expect(errorHandlerStub.record).to.have.been.calledWith("error", functionName, "create");
+    });
+  });
+
+  describe("schedulerDeploymentHandler", () => {
+    const sandbox = sinon.createSandbox();
+    let errorHandlerStub: sinon.SinonStubbedInstance<ErrorHandler>;
+
+    beforeEach(() => {
+      errorHandlerStub = sandbox.createStubInstance(ErrorHandler);
+    });
+
+    afterEach(() => {
+      sandbox.restore();
+    });
+
+    it("should execute the task", async () => {
+      const run = sinon.spy();
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: "myFunc",
+        operationType: "upsert schedule",
+      };
+
+      const handler = tasks.schedulerDeploymentHandler(errorHandlerStub);
+      await handler(testTask);
+
+      expect(run).to.have.been.called;
+      expect(errorHandlerStub.record).not.to.have.been.called;
+    });
+
+    it("should throw quota errors", async () => {
+      const run = sinon.spy(() => {
+        throw new FirebaseError("an error occurred", {
+          status: 429,
+        });
+      });
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: "myFunc",
+        operationType: "upsert schedule",
+      };
+
+      const handler = tasks.schedulerDeploymentHandler(errorHandlerStub);
+      await expect(handler(testTask)).to.eventually.be.rejected;
+
+      expect(run).to.have.been.called;
+      expect(errorHandlerStub.record).not.to.have.been.called;
+    });
+
+    it("should ignore 404 errors", async () => {
+      const run = sinon.spy(() => {
+        throw new FirebaseError("an error occurred", {
+          status: 404,
+        });
+      });
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: "myFunc",
+        operationType: "upsert schedule",
+      };
+
+      const handler = tasks.schedulerDeploymentHandler(errorHandlerStub);
+      await handler(testTask);
+
+      expect(run).to.have.been.called;
+      expect(errorHandlerStub.record).not.to.have.been.called;
+    });
+
+    it("should handle other errors", async () => {
+      const run = sinon.spy(() => {
+        throw new FirebaseError("an error occurred", {
+          status: 500,
+        });
+      });
+      const functionName = "myFunc";
+      const testTask: tasks.DeploymentTask = {
+        run,
+        functionName: functionName,
+        operationType: "upsert schedule",
+      };
+
+      const handler = tasks.schedulerDeploymentHandler(errorHandlerStub);
+      await handler(testTask);
+
+      expect(run).to.have.been.called;
+      expect(errorHandlerStub.record).to.have.been.calledWith(
+        "error",
+        functionName,
+        "upsert schedule"
+      );
+    });
+  });
+});