A method for securing an online service via a continuously learning model, comprising: gathering a set of security signals from the online service, wherein the set of security signals are gathered in a rolling window of time; examining each security signal of the set of security signals via predictive models to identify whether each security signal is malicious or benign, wherein the predictive models are configured to produce a detection result of whether a given security signal is malicious or benign based on a feature vector defined by an administrative user; associating the security signals with the detection results to identify the security signals as malicious signals or benign signals; balancing the malicious signals with the benign signals to produce a balanced training dataset, including: identifying an attack type of each of the malicious signals; identifying a device type from which each of the benign signals was gathered; equalizing relative numbers of malicious signals in the rolling window based on identified attack types to produce a set of attack examples; equalizing relative numbers of benign signals in the rolling window based on identified device types to produce a set of benign examples identified; and cross joining the set of attack examples with at least a portion of the set of benign examples to balance a number of attack examples in the set of attack examples relative to a number of benign examples in the set of benign examples; and refining the predictive models based on the balanced training dataset and a machine learning algorithm. gathering a set of security signals from the online service, wherein the set of security signals are gathered in a rolling window of time; examining each security signal of the set of security signals via predictive models to identify whether each security signal is malicious or benign, wherein the predictive models are configured to produce a detection result of whether a given security signal is malicious or benign based on a feature vector defined by an administrative user; associating the security signals with the detection results to identify the security signals as malicious signals or benign signals; balancing the malicious signals with the benign signals to produce a balanced training dataset, including: identifying an attack type of each of the malicious signals; identifying a device type from which each of the benign signals was gathered; equalizing relative numbers of malicious signals in the rolling window based on identified attack types to produce a set of attack examples; equalizing relative numbers of benign signals in the rolling window based on identified device types to produce a set of benign examples identified; and cross joining the set of attack examples with at least a portion of the set of benign examples to balance a number of attack examples in the set of attack examples relative to a number of benign examples in the set of benign examples; and identifying an attack type of each of the malicious signals; identifying a device type from which each of the benign signals was gathered; equalizing relative numbers of malicious signals in the rolling window based on identified attack types to produce a set of attack examples; equalizing relative numbers of benign signals in the rolling window based on identified device types to produce a set of benign examples identified; and cross joining the set of attack examples with at least a portion of the set of benign examples to balance a number of attack examples in the set of attack examples relative to a number of benign examples in the set of benign examples; and refining the predictive models based on the balanced training dataset and a machine learning algorithm.