Using Your Data to Stop Credit Card Fraud: Capital One and Other Best Practices - The Databricks BlogSkip to main contentPlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWCategoriesAll blog postsCompanyCultureCustomersEventsNewsPlatformAnnouncementsPartnersProductSolutionsSecurity and TrustEngineeringData Science and MLOpen SourceSolutions AcceleratorsData EngineeringTutorialsData StreamingData WarehousingData StrategyBest PracticesData LeaderInsightsIndustriesFinancial ServicesHealth and Life SciencesMedia and EntertainmentRetailManufacturingPublic SectorUsing Your Data to Stop Credit Card Fraud: Capital One and Other Best Practicesby Fahmid KabirJuly 13, 2021 in Company BlogShare this postFraud is a costly and growing problem – research estimates that $1 of fraud costs companies 3.36x in chargeback, replacement and operational cost. Adding to the pain, according to experts, there are not enough regulations to protect small businesses from chargebacks and losses from fraud. Despite significant advancements in credit card fraud, risk management techniques have adapted, and fraudsters are still able to find loopholes and exploit the system. For credit card companies, the threat of fraudulent card usage is a constant, which results in the need for accurate credit card fraud detection systems. All organizations are at risk of fraud and fraudulent activities, but that risk is especially burdensome to those in financial services. “Threats can originate from internal or external sources, but the effects can be devastating - including loss of consumer confidence, incarceration for those involved, and even the downfall of corporations,” says Badrish Davay, a Data Engineering and Machine Learning leader at Capital One. CNBC reports that the US is the most credit card fraud prone country in the world.
Fraud detection using machine learning 
It’s not all bad news, though. With modern advancements, businesses are able to stay ahead of threats by leveraging data and machine learning. As part of a tech talk at the recent Data + AI Summit, we were able to get a glimpse into how Capital One is using data and artificial intelligence (AI) to address fraud. Badrish Davay from Capital One shared how we can utilize state-of-the-art ML algorithms to stay ahead of the attackers and, at the same time ,constantly learn new ways a system is being exploited. “In order to more dynamically detect fraudulent transactions, one can train ML models on a dataset, including credit card transaction data, as well as card and demographic information of the cardholder. Capital One uses Databricks to achieve this goal,” noted Davay.
To play this video, click here and accept cookiesCapital One analyzes all the fraudulent activities to understand what to look for in credit card fraud. Davay presented the 6 “W” questions they ask –  what, who, when, where, why and what if? – used to uncover trends in fraudulent activities. Davay highlighted various scenarios in which card information may be compromised and how data can help with anomaly detection and identifying fraud. For example, he shared how geospatial data can detect stolen card information when it is being used away from its actual location, along with temporal data to determine fraud.
As Davay also explained, when a customer physically loses a card, but doesn’t notify the organization,  contextual information ( e.g. work hours, spending habits, etc.) can help determine if transactions are routine or anomalous. A key takeaway from Davay is that we should be able to combine a multiple of independent signals together to get a wider context around traction and demographics data. With the availability of data and advancements in ML, fraud prevention is a key area in which ML is changing both workflows and outcomes, allowing organizations to stay ahead of increasingly technologically advanced criminals.
Today’s businesses are facing an increasingly sophisticated enemy that attacks, responds and changes tactics extremely quickly. Due to dynamics of fraud, organizations need AI to constantly adapt to changing behaviours and patterns. AI brings agility that rules do not. With data analytics and ML, companies can get ahead of threats. Below are some key reasons why ML is apt for taking on fraud:
Fraud hides under massive amounts of data: The most effective way to detect fraud is to look at the overall behaviors of end users. Looking at transactions or orders is not enough — we need to follow the events leading up to and after the transaction. This culminates into a lot of structured and unstructured data, and the best way to detect fraud in such huge volumes is with ML and AI.Fraud happens quickly: When an ML system updates in real time, that knowledge can be used within milliseconds to update fraud detection models and prevent an attack.Fraud is always changing: Fraudsters constantly adapt their tactics, making them difficult for humans to detect  – and impossible for static rules-based systems, which don’t learn. ML, however, can adapt to changing behavior.Fraud looks fine on the surface: To the human eye, fraudulent and normal transactions don’t appear any differently from each other. ML has a deeper and more nuanced way of viewing data, which helps avoid false positives.Davay discussed how MLuses statistical models, such as classifiers, and logistic regression to look at past outcomes and anomalies to predict future outcomes. An ML system can learn, predict and make decisions as data comes in real time. In his presentation, Davay outlined what a good fraud prevention model needs to have:
one-stop shop for users to train the model and orchestrate executionReal-time detectionDeep analytics and  modeling by leveraging powerful ML tools, such as deep learning and neural networks, for what if data analysis and testing new hypothesesAdherence to company security policy and compliance requirementsNotification service  to inform cardholders immediately of suspicious activitySeamless integration with enterprise systemsMLflow in fraud prevention 
Davay highlighted the value of Databricks and MLflow in their fraud prevention efforts. He talked about the platform and how different data and fraud teams collaboratively develop and run experiments with the team using Databricks. “Even though they share experiments and data collaboratively within the team, we can implement stringent security measures in order to respect data privacy, and each experiment can have its own compute environments and requirements,” said Davay.  He referred to Databricks as “a one-stop shop for all of [their] data science and models, making it perfect for data science projects.” When the team has identified features for predicting whether a transaction is fraudulent or not, they pass these data points to Databricks’ hosted environment, where they can then perform feature engineering, data pre-processing and split the data into test and training sets. They then use a variety of supervised or unsupervised ML algorithms, such as SVM, decision tree and random forest, to train a model. They identify the best performing model and use the Databricks Lakehouse Platform to solve for fraud directly from within the platform. The lakehouse is a conducive environment for fraud detection and you can learn more from our solution accelerators here.

Davay mentioned how “MLflow within the Databricks ecosystem is a great feature that we can use because it has numerous advantages in developing the ML workflow pipeline seamlessly.” MLflow allows Capital One to track their ML experiments from end-to-end throughout the ML model lifecycle. During the talk, Davay mentioned they can run experiments directly from GitHub without the need to go through the code and can directly deploy and train models by serializing them while utilizing packages such as Python’s pickle module, Apache Spark, and  MLflow. They then deploy the serialized model and serve it as an API by harnessing MLflow.
MLflow and microservices
Davay also touched on microservices and why they are useful in MLflow. A microservice is a gateway to a specific functional aspect of an application. It helps teams like Capital One’s develop applications in a standardized consistent manner over time. Microservices allow Capital One to deploy functionality of applications independent of each other. It helps abstract the functionality while enabling the team to build in a reusable and uniform way of interacting with an application. Furthermore, it lets teams compose complex behavior by combining a variety of other microservices together. Essentially, it empowers companies to use any tech stack in the backend while maintaining compatibility on the front end.
With Capital One’s raw data stored in Amazon S3, they quickly integrate interactions between S3 and their framework through Databricks seamlessly and can massively scale ML  model training, validation and deployment pipelines through MLflow. Their team trains and validates models on custom clusters in AWS and deploys them through SageMaker directly by using MLflow APIs. MLflow is not only limited to AI, but can embed any piece of business logic (as mentioned in Databricks Rules + AI accelerator) and, as such, benefits from the E2E governance and delivery principles as microservices.
Putting it all together 
Davay shared how Databricks allows Capital one to query and deploy models and manage and clean up the deployment while using MLflow APIs within the AWS ecosystem. In addition, they can ensure safe security and conditional access via AWS SSO.
Based on observations from Capital One and many other customers, there are several benefits of using data and AI for fraud prevention, including:
Reduced need for manual review. ML automates processes in which behaviors can be learned at the individual level and  detect anomalies.The ability to prevent fraud cases without impeding the user experience. AI brings automation to the process seamlessly and prevents fraud in advance without burdening users.Lower operational costs than other approaches. With less manual work and automation, data and AI require fewer resources and preempts losses associated with fraud.Frees up teams’ time to focus on more strategic tasks. Most companies are not in the business of fraud detection, and an ML fraud prevention process can help them focus on core activities.Adapts quickly. Coupled with human talent and experience, data and AI work together to constantly learn and adjust to new user behaviors and trends.When it comes to operationalizing data and AI to build customer relationships and drive higher returns on equity, fraud should be considered a top priority. Curbing fraudulent or malicious behavior  – from fraudulent card transactions – is key to mitigating negative revenue impact. To more dynamically detect fraudulent transactions, Capital One uses ML and credit card transaction information, as well as card and demographic information, to get a comprehensive view to identify anomalies. Data-driven innovators such as Capital One’s are paving the way in fraud detection and provide a successful model to follow to protect customers and business.
Get started with fraud prevention in DatabricksGet a jump start with our our prebuilt code and guides in our Fraud Solution Accelerators See all our Financial Services solutionsTry Databricks for freeGet StartedRelated postsUsing Your Data to Stop Credit Card Fraud: Capital One and Other Best PracticesJuly 13, 2021 by Fahmid Kabir in Company Blog
Fraud is a costly and growing problem – research estimates that $1 of fraud costs companies 3.36x in chargeback, replacement and operational cost. A...
See all Company Blog postsProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights