Reference number : GC-2017-Q2-00339, amendment number : , agreement type : G, recipient legal name : Beschastnikh, Ivan (The University of British Columbia), recipient country : CA, recipient province : nan, recipient city : British Columbia|Colombie-Britannique, recipient postal code : nan, program name english : Engage Grants for Universities, program name french : Subventions d'engagement partenarial pour les universités, program purpose english : Big Data processing frameworks allow enterprises to drive their business based on key insights extracted from_x000D_
near real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customer_x000D_
information and expose data insights to the shop owners who are users of the Shopify platform. For example,_x000D_
the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform to_x000D_
learn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) by_x000D_
geographical region or by age group, and see if the marketing campaign resulted in actual product sales._x000D_
Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists of_x000D_
multiple jobs, each applying some computation to the data. A set of connected jobs form a flow which might_x000D_
correspond to the query exposed to the shop owner. In this project we will explore algorithmic solutions to_x000D_
maintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuring_x000D_
co-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approaches_x000D_
based on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best in_x000D_
practice., program purpose french : Big Data processing frameworks allow enterprises to drive their business based on key insights extracted from_x000D_
near real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customer_x000D_
information and expose data insights to the shop owners who are users of the Shopify platform. For example,_x000D_
the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform to_x000D_
learn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) by_x000D_
geographical region or by age group, and see if the marketing campaign resulted in actual product sales._x000D_
Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists of_x000D_
multiple jobs, each applying some computation to the data. A set of connected jobs form a flow which might_x000D_
correspond to the query exposed to the shop owner. In this project we will explore algorithmic solutions to_x000D_
maintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuring_x000D_
co-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approaches_x000D_
based on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best in_x000D_
practice., agreement title english : Optimizing compute task scheduling at Shopify, agreement title french : Optimizing compute task scheduling at Shopify, agreement number : EGP, agreement value : 25000.0, agreement start date : 2017-08-23, agreement end date : nan, description english : nan, description french : nan, expected results english : nan, expected results french : nan, additional information english : Grant or Award spanning more than one fiscal year (2017-2018 to 2018-2019)., additional information french : Subvention ou bourse octroyée s'appliquant à plus d'un exercice financier (2017-2018 à 2018-2019)., owner organization : nserc-crsng, owner organization title : Natural Sciences and Engineering Research Council of Canada | Conseil de recherches en sciences naturelles et en génie du Canada