Title: Scalable Feature Learning on Huge Knowledge Graphs for Downstream Machine Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge graph

Downstream tasks

Scalable

Feature learning
Abstract: Many machine learning tasks can benefit from external knowledge. Large knowledge graphs store such knowledge, and embedding methods can be used to distill it into ready-to-use vector representations for downstream applications. For this purpose, current models have however two limitations: they are primarily optimized for link prediction, via local contrastive learning, and their application to the largest graphs requires significant engineering effort due to GPU memory limits. To address these, we introduce SEPAL: a Scalable Embedding Propagation ALgorithm for large knowledge graphs designed to produce high-quality embeddings for downstream tasks at scale. The key idea of SEPAL is to ensure global embedding consistency by optimizing embeddings only on a small core of entities, and then propagating them to the rest of the graph with message passing. We evaluate SEPAL on 7 large-scale knowledge graphs and 46 downstream machine learning tasks. Our results show that SEPAL significantly outperforms previous methods on downstream tasks. In addition, SEPAL scales up its base embedding model, enabling fitting huge knowledge graphs on commodity hardware. Our code is available at: <https://github.com/soda-inria/sepal>.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProposes a method to obtain knowledge graph embeddings for large knowledge graphs. This is done by decomposing the graph into a \"core\" (used for training and to obtain embeddings of frequent entities) and multiple potentially overlapping \"rest\" subgraphs (updated via message passing to obtain embeddings of remaining entities). Reports on an experimental study.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength of this work lies in it's simplicity:\n\nS1. Simple method\n\nS2. Cheap to run\n\nS3. Reasonable heuristic (most frequent entities + neighbors)\n\nThe approach as a whole, however, fails to make a convincing case:\n\nW1. Cost arguments not convincing. KGE methods have been scaled to graphs larger than the largest one considered in the experimental study, including many works mentioned in Sec. 2.3 and including works that require few resources. It's not clear if there is a need for another method.\n\nW2. The subgraph propagation part (which is the key technical contribution) is not convincing. This is for three reasons:\n\nW2-1: Analysis limited. For the subgraph propagation step, the authors claim in text that their approach minimizes \"a global energy\". The actual proposition then limits this to DistMult, identity similarity and \"sum of scores\" objective; this is does not reflect an actually interesting setting (e.g., sum of scores is not a suitable KGE training objective).\n\nW2-2: The method seems to be equivalent to the one of Albooyeh et al. (EMNLP Findings, 2020, https://aclanthology.org/2020.findings-emnlp.241/).\n\nW3-3: It's not clear why the decomposition method (appendix) is needed for propagation and what its impact is w.r.t. to the analysis of Prop. 4.1. Can't this simply be done on the CPU (e.g., stream edges through)?\n\nW3. Experimental setup not clear. Neither the hyperparameters (which ones there are and which ranges have been explored) nor the search strategy has been described in the paper. This includes hyperparameters of the proposed method as well as the underlying KGE method, and the cost/validation objectives used for the core graph.\n\nW4. Experimental study not insightful. This is for multiple reasons.\n\nW4-1: The paper compares to embeddings trained using PyTorch BigGraph and DGL-KE, neither of which is a state of the art method; i.e., no newer \"large KG\"-methods are considered.\n\nW4-2: Transductive methods are not timely anymore, esp. in the KG setting, since good inductive methods exist by now; e.g., Cui et al. (NeurIPS 2024, https://proceedings.neurips.cc/paper_files/paper/2024/hash/0d70af566e69f1dfb687791ecf955e28-Abstract-Conference.html). Such methods are neither discussed nor compared to.\n\nW4-3: The proposed method aims to optimize link prediction performance cheaply (as that's the idea behind propagation), and, as expected, falls behind link prediction methods (as shown in the appendix). Nevertheless, the proposed method achieves better vertex classification/regression scores in the experimental study. It's neither explored nor intuitively clear why this is the case.\n\nQUESTIONS:\n/\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
SEPAL is an efficient method for large-scale knowledge graph embedding, addressing computational and memory bottlenecks through message passing and the BLOCS algorithm. Its improvements include:Time optimization: Utilizing the message passing mechanism, the embeddings of the core subgraph (selected by degree or a hybrid strategy of highly connected entities) are propagated to the entire graph, preserving the geometric properties of the relationships.Memory optimization: The graph is divided into overlapping subsets via the BLOCS algorithm to accommodate GPU memory constraints. The selection of the core subgraph takes into account both entity density and relationship diversity (such as a hybrid strategy combining node degree and relationship types), ensuring the quality of the embeddings.

STRENGTHS AND WEAKNESSES:
Strengths: 
1.Significantly improve the performance of downstream tasks 

2.A single 32GB V100 GPU can handle graphs with hundreds of millions of nodes (such as the 91 million entities in WikiKG90Mv2), while baseline methods (such as DGL-KE) require distributed computing. 

3. It can capture the global structure and enhance the geometric consistency of the embedding by replacing local contrastive learning with relation-aware propagation.

Weaknesses:
1.Experimental design flaw: Insufficient coverage of downstream tasks
Most tasks are based on table regression/classification and have not been validated in complex scenarios such as multi hop inference.
2.In WikiKG90Mv2 link prediction, SEPAL's MRR (0.195) is significantly lower than PBG's (0.632).

QUESTIONS:
See Weakness.

LIMITATIONS:
Yes, the proposed method, while promising, reveals certain limitations in the Experiments section.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
Knowledge graphs are widely used to store knowledge and can be leveraged to produce embeddings that benefit various applications. However, existing methods usually do not scale well. This paper proposes a method that first obtains embeddings on a small subgraph called core subgraph and then uses message passing to propagate embeddings outside the core subgraph. Extensive experiments corroborate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1. The considered problem is of great importance to both academia and industry.

2. The paper is well-written and easy-to-follow.

3. The experimental results are sufficient to prove the effectiveness of the proposed method, from my perspective.

4. The propagation step outside the core subgraph enjoys the advantage of existing knowledge graph embedding models and further affirms their effectiveness.

5. The propagation step is theoretically guaranteed.

**Weaknesses**:

1. The BLOCS algorithm is described in the appendix. The authors could still describe its high-level ideas and main steps in the main paper.

2. The authors mainly determine core subgraphs using node degrees. Other node centrality measures like PageRank score could also be tried.

QUESTIONS:
See above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two things – an embedding propagation algorithm called SEPAL that scales to large KGs and BLOCS, an algorithm to break large graphs into balanced overlapping subgraphs. The algorithm works in two phases – identifying a core subgraph and creating core and relation embeddings using existing KGE methods and propagating the embeddings to the other subgraphs using message passing. The authors show that this process can be executed in a limited GPU memory setting for really large graphs. The evaluation on downstream applications shows cases the merit of the proposed solution.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The authors have clearly defined the problem and where the proposed solution comes in. \nS2: The proposed algorithm SEPAL reuses several methods from the existing graph / KG literature towards scaling of the KGE generation, and the authors also provide theoretical analysis for SEPAL.\nS3: The proposed algorithms are evaluated on a broad set of KGs and of varying lengths showcasing the strength of the solution in terms of usage of only commodity hardware to compute the embeddings.\nW1: The biggest weakness is that the paper is heavily reliant on the Appendix and while it is appreciated that the authors have provided substantial and important information in the appendix, the main paper should be representative of the core work (both algorithms and the evaluation). In the current format, it takes a significant amount of back and forth to understand the work.\nW2: (Minor) The placement of the proposed algorithm BLOCS in appendix is slightly confusing as it is claimed to be a contribution but only mentioned in the main paper.\n\nQUESTIONS:\nQ1: If the knowledge graph is updated with new entities and relations, does the entire process need to rerun (starting from code graph selection)?\nQ2: Is there a dependency on the graph having higher connectivity or connectedness for the SEPAL algorithm?\n\nLIMITATIONS:\nAuthors have sufficiently addressed the limitations of the work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}