Reference number : GC-2017-Q3-00370, amendment number : , agreement type : G, recipient legal name : Krishnamurthy, Diwakar (University of Calgary), recipient country : CA, recipient province : nan, recipient city : Alberta, recipient postal code : nan, program name english : Collaborative Research and Development Grants - Project, program name french : Subventions de recherche et développement coopérative - projet, program purpose english : The proposed project will develop tools that will allow users and operators of big data systems to better analyze and optimize the performance of their applications. Enterprises are increasingly relying on big data cluster systems to support their analytics requirements. Newer generation big data platforms such as Apache Spark provide a rich set of Application Programming Interfaces (APIs) that allow data scientists to quickly develop and deploy analytics applications. However, despite recent advances in big data platforms, analyzing and optimizing the performance of big data applications remains a challenge. Users and operators of big data clusters often face challenges in understanding the reasons for poor application performance and optimizing applications and clusters to meet performance requirements. _x000D_
To address these challenges, the proposed project will focus on two long term objectives. First, the project will devise an intuitive and intelligent interface that will support rapid performance analysis and optimization of big data applications. Using the interface, a user will be able to quickly identify bottlenecks that limit an application's performance. The interface can also be used to visualize and understand an application's performance scalability. Such features can in turn lead to increased productivity for users and more efficient usage of clusters for operators. Second, the project will develop automated service level management techniques for big data systems. Typically, big data systems use schedulers to control the allotment of resources to applications. Although production scheduling systems are sophisticated, they currently lack the ability to drive scheduling decisions so as to automatically meet user service level requirements, e.g., a specified deadline for completing an application, while simultaneously achieving efficient resource usage. Consequently, users and cluster operators often resort to tedious trial and error approaches, which can cause both missed service levels as well as inefficient resource usage. The proposed project will devise new scheduling techniques to address this limitation. _x000D_
_x000D_
_x000D_, program purpose french : The proposed project will develop tools that will allow users and operators of big data systems to better analyze and optimize the performance of their applications. Enterprises are increasingly relying on big data cluster systems to support their analytics requirements. Newer generation big data platforms such as Apache Spark provide a rich set of Application Programming Interfaces (APIs) that allow data scientists to quickly develop and deploy analytics applications. However, despite recent advances in big data platforms, analyzing and optimizing the performance of big data applications remains a challenge. Users and operators of big data clusters often face challenges in understanding the reasons for poor application performance and optimizing applications and clusters to meet performance requirements. _x000D_
To address these challenges, the proposed project will focus on two long term objectives. First, the project will devise an intuitive and intelligent interface that will support rapid performance analysis and optimization of big data applications. Using the interface, a user will be able to quickly identify bottlenecks that limit an application's performance. The interface can also be used to visualize and understand an application's performance scalability. Such features can in turn lead to increased productivity for users and more efficient usage of clusters for operators. Second, the project will develop automated service level management techniques for big data systems. Typically, big data systems use schedulers to control the allotment of resources to applications. Although production scheduling systems are sophisticated, they currently lack the ability to drive scheduling decisions so as to automatically meet user service level requirements, e.g., a specified deadline for completing an application, while simultaneously achieving efficient resource usage. Consequently, users and cluster operators often resort to tedious trial and error approaches, which can cause both missed service levels as well as inefficient resource usage. The proposed project will devise new scheduling techniques to address this limitation. _x000D_
_x000D_
_x000D_, agreement title english : Performance management tools for big data applications, agreement title french : Performance management tools for big data applications, agreement number : CRDPJ, agreement value : 68000.0, agreement start date : 2017-10-18, agreement end date : nan, description english : nan, description french : nan, expected results english : nan, expected results french : nan, additional information english : Grant or Award spanning more than one fiscal year (2017-2018 to 2018-2019)., additional information french : Subvention ou bourse octroyée s'appliquant à plus d'un exercice financier (2017-2018 à 2018-2019)., owner organization : nserc-crsng, owner organization title : Natural Sciences and Engineering Research Council of Canada | Conseil de recherches en sciences naturelles et en génie du Canada