Project: Vineyard (Runtime / Cloud Native Storage)
File: 2023-vineyard-annual.md

is an in-memory immutable data manager that provides out-of-the-box high-level abstraction and zero-copy in-memory sharing for distributed data in big data tasks, such as graph analytics, numerical computing, and machine learning. Vineyard is design to address the inefficiency of data sharing in big data analytical workflows on Kubernetes. Vineyard provides: Efficient in-memory data management and zero-copy sharing across different systems. Out-of-the-box high-level data abstraction for distributed objects (e.g., tensors, tables, graphs, and distributed datasets) and efficient polyglot support (currently including C++, Python, Go, Rust and Java). Seamless integration with Kubernetes for cluster deployment and management, workloads orchestration, and observability. Out-of-the-box integration with workflow orchestration engines (including , and ), providing end-users with a unified and intrusive experience to leverage Vineyard in their data-intensive workflows to improving performance. Alignment with CNCF: Vineyard builds on Kubernetes for deploying and scaling, and the objects are observable in Kubernetes as CRDs. Vineyard makes efficient zero-copy sharing possible for data-intensive workflows on cloud-native infrastructure by a data-aware Kubernetes scheduler plugin. Vineyard adopts an immutable object design, which aligns with the immutable infrastructure of the cloud-native environment. Vineyard aligns with the CNCF effort on helping migrate batching system workflows to cloud native environments. Include a link to your projects devstats page. We will be looking for signs of consistent or increasing contribution activity. Please feel free to add commentary to add colour to the numbers and graphs we will see on devstats. Stargazers and Forks Commits per week Contributors and Companies The vineyard community has grown since the project entered the CNCF sandbox. Number of contributors: 26 -> 40 Github stars: 600+ -> ~750 Github forks: 80+ -> 110+ Vineyard published 8 release (one release about per 1.5 month) since the last annual review. The major new features and improvements include: Language SDKs in Rust and Go, where the Rust SDK was a collaboration with our external end-user, and enabled users seamlessly and efficiently interoperating their data between Python and Rust. Integration with the workflow engine Kedro, and gained attention in the Kedro community. Vineyard supports the data processing engine, letting users can easily connect traditional data processing pipelines built with the Hadoop ecosystem with emerging big-data and AI applications (e.g., applications in the community). A initial version of CSI driver, which helped Vineyard aligned with the Kubernetes platform and enables users to leverage Vineyard in their Kubeflow pipelines to optimize the data sharing between steps with only minor changes to their existing source code. We have conducted a series of research work around Vineyard and published the paper Vineyard: Optimizing Data Sharing in Data-Intensive Analytics in SIGMOD 2023, a top-tier conference in the data management community. Wenyuan Yu, Tao He, Lei Wang, Ke Meng, Ye Cao, Diwen Zhu, Sanhong Li, Jingren

Vineyard: Optimizing Data Sharing in Data-Intensive Analytics. ACM SIG Conference on Management of Data (SIGMOD), industry, 2023. . How many maintainers do you have, and which organisations are they from? (Feel free to link to an existing MAINTAINERS file if appropriate.) We currently have 10 maintainers and 2 committers and have . Initial maintainers | Name | GitHub ID | Affiliation | Email | | | | | | | Tao He | | Alibaba | | | Xiaojian Luo | | Alibaba | | | Ke Meng | | Alibaba | | | Wenyuan Yu | | Alibaba | | | Weibin Zeng | | Alibaba | | | Siyuan Zhang | | Alibaba | | | Diwen Zhu | | Alibaba | | New maintainers in this year | Name | GitHub ID | Affiliation | Email | | | | | | | Ye Cao | | Alibaba | | | Shumin Yuan | | Alibaba | | | Denghao Li | | PingAn Tech | | New Committers in this year | Name | GitHub ID | Affiliation | Email | | | | | | | Lihong Lin | | PKU | | | Pei Li | | CMU | | What do you know about adoption, and how has this changed since your last review / since you joined Sandbox? If you can list companies that are end-users of your project, please do so. (Feel free to link to an existing ADOPTERS file if appropriate.) We have tracked the following two major adoption since StartDT (Qidianyun): transiting towards production stage StartDT is a startup company in China, providing a cloud-native data platform for big-data analytics and machine learning applications. Vineyard is currently used in their Python-centric data processing pipelines to share distributed dataframe artifacts between steps, and help build a composable and efficient data processing platform to end-users. Vineyard has passed their eager-evaluation, and they are working on building their distributed data processing platform on top of Vineyard. PingAn Tech: production stage PingAn is a large-scale fin-tech company in China. Vineyard is used in their data science platform to support efficient dataset sharing and management among data science researchers. The status of Vineyard in their platform has been transited from testing to production stage, and one of their engineers has become a maintainer of the Vineyard project. Besides these two major companies, since our last annual review, we have also noticed some other questions about using Vineyard in machine learning inference scenarios, but we haven't tracked the actual adoption yet. How has the project performed against its goals since the last review? (We won't penalize you if your goals changed for good

Vineyard has successfully archived the goals about easing the getting started process for end-users from three aspects: Out-of-the-box integration with data processing systems, especially Spark and Hive, the most popular data processing engines in the big data community; Data processing pipeline orchestration: providing non-intrusive interfaces to help users migrate their existing data processing pipelines to Vineyard on Kubernetes and finally benefit from the efficient data sharing; Seamless inter-operability with other systems in the cloud-native environments: we invest a lot of effort in the Vineyard operator to help use deploy vineyard along with their workloads in a non-intrusive, declarative way and has tested the functionality with GraphScope in end-users production environments. Besides, Vineyard has successfully attracted new end-users from the big data community to adopt Vineyard in their own data processing platform, and the feedback from the Kedro community is also positive. What are the current goals of the project? For example, are you working on major new features? Or are you concentrating on adoption or documentation? Our current goals are mainly focused on the attracting more end-user to adopt Vineyard in their scenarios from different domains. Specifically, we are keeping moving towards the following goals in the next year: Optimizing our current Kubeflow integration and find more opportunities to evaluate and deploy Vineyard in production machine learning applications; Publish our integration with the big data processing systems to their end-user community and gather feedback for further improvements; Seeking more opportunities to evaluate Vineyard in the emerging LLM applications, for both data preprocessing, training, and inference serving to see if Vineyard can bring added value to these applications as where the data cost is usually high; Getting engaged with the Batch System WG in CNCF to seek opportunities about further collaboration with other projects in CNCF. How can the CNCF help you achieve your upcoming goals? Vineyard has incredibly benefited from CNCF since accepted as a sandbox project. We believe the end-users in the CNCF community are critical for Vineyard to become successful. With the help of CNCF service desk, we have successfully built a new website for Vineyard, which is more friendly to end-users. We are also working on components like CSI driver and hope that could make the inter-operation with other projects in the CNCF community easier. We will host a Project Kiosk in this KubeCon China and hope to get more feedback from the community, and hope to get more feedback from the community. Furthermore, we hope we could have more opportunities to introduce our project to border end-users in the CNCF community to increase adoption. Do you think that your project meets the ? We think our project vineyard still needs further exploration to get border adoption in the end-user's production deployment and gather more feedback, and we are looking forward to meeting the incubation criteria in the near future.