Title: Differentially Private Relational Learning with Entity-level Privacy Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: Differential Privacy

Relational Learning

Privacy Amplification
Abstract: Learning with relational and network-structured data is increasingly vital in sensitive domains where protecting the privacy of individual entities is paramount. Differential Privacy (DP) offers a principled approach for quantifying privacy risks, with DP-SGD emerging as a standard mechanism for private model training. However, directly applying DP-SGD to relational learning is challenging due to two key factors: (i) entities often participate in multiple relations, resulting in high and difficult-to-control sensitivity; and (ii) relational learning typically involves multi-stage, potentially coupled (interdependent) sampling procedures that make standard privacy amplification analyses inapplicable. This work presents a principled framework for relational learning with formal entity-level DP guarantees. We provide a rigorous sensitivity analysis and introduce an adaptive gradient clipping scheme that modulates clipping thresholds based on entity occurrence frequency. We also extend the privacy amplification results to a tractable subclass of coupled sampling, where the dependence arises only through sample sizes. These contributions lead to a tailored DP-SGD variant for relational data with provable privacy guarantees. Experiments on fine-tuning text encoders over text-attributed network-structured relational data demonstrate the strong utility-privacy trade-offs of our approach.


Peer reviews:
comments: SUMMARY:
A fundamental problem that limits the DP-SGD in relational learning is that DP-SGD considers the i.i.d assumption of samples when clipping the gradient and mini-batching. In relational learning, the samples, i.e., the nodes and edges, are dependent. To mitigate this problem, the authors design a new DP-SGD with a specifically designed gradient-clipping strategy and mini-batch construction strategy in this paper.

STRENGTHS AND WEAKNESSES:
Strength:
1. This paper has good presentation quality.

2. The analysis of the problem and the intuition of new methods is comprehensive and makes sense to me.

3. The problem studied by the paper is vital for protecting training data privacy in relational learning.


Weakness:

1. I think the authors should publish their code to increase the reproducibility.

2. Some questions are not answered, please check the questions part.

QUESTIONS:
1. I think the adaptive clipping scheme relies on per-entity frequency, but the paper does not analyse how the joint distribution of frequencies affects the optimal clip threshold once entities number in the millions (e.g., social graphs). A worst-case entity that appears in most relations could still dominate sensitivity—even with adaptive scaling—forcing very large noise that may erase the reported utility gains. It's might be too complex to present in a single paper, but I'm curious about how the authors think about this problem.

2. Entity-level DP is stronger but costlier than edge-level DP. I'm curious about how much additional noise is paid relative to edge-level privacy.

LIMITATIONS:
No.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a DP-SGD framework for relational learning with formal node-level differential privacy guarantees. To address high sensitivity and coupled sampling challenges, the authors introduce adaptive gradient clipping based on node frequency and a new privacy amplification analysis for cardinality-dependent sampling. Experiments on fine-tuning text encoders demonstrate improved privacy–utility trade-offs over standard DP-SGD.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- The paper proposes adaptive gradient clipping and a new amplification analysis tailored for relational learning, offering node-level differential privacy guarantees.\n- The proposed method achieves strong privacy–utility trade-offs across various datasets.\n- This paper is clearly written, and the tables and charts are easy to understand.\n\nWeaknesses\n- The work feels somewhat related to the well-established problem of differentially private link prediction. A discussion highlighting the differences and connections with existing literature on DP link prediction would help clarify the unique contribution of this approach.\n- The compared baselines (standard clipping) are limited. Adapting additional recent baselines for comparison would provide a more comprehensive evaluation of the efficacy of the proposed method.\n\nQUESTIONS:\n- This work focuses on node-level DP. Does it offer any insights or potential solutions for achieving edge-level DP in the same context?\n- Why is fine-tuning pretrained text encoders chosen as the application? In the task of relational learning, or more specifically, link prediction, GNNs are often directly applied. Is this method suitable for such cases as well?\n- In scenarios like online social platform recommendations, where the number of high-degree nodes can be very large, with degrees reaching into the hundreds or thousands, how does degree capping influence performance in such cases?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a principled framework for differentially private relational learning, focusing on edge existence as the sensitive attribute. It introduces adaptive gradient clipping and a novel sampling analysis to handle high sensitivity and interdependent data and uses these to introduce an new DP-SGD framework suited to relational learning.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper addresses a timely and previously unexplored problem, contributing novel insights to the field.

- It presents a comprehensive critique of existing approaches, clearly articulating their limitations. The proposed framework demonstrates significantly improved privacy guarantees compared to naive baselines.

Weaknesses:
-The experimental evaluation lacks results for high-privacy regimes (e.g., epsilon = 1), which limits the assessment of the framework's performance under stricter privacy constraints.

QUESTIONS:
- Would the performance improvement be as significant in the high privacy regime?
- Can you provide an upper bound for ε(α) in the theorem to help with its interpretation?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for relational learning with entity-level differential privacy (DP) guarantees, addressing challenges unique to graph-structured data.  Relational learning, which leverages relationships between entities, is increasingly applied in sensitive domains like healthcare and finance, where privacy concerns are paramount.  Standard DP mechanisms, such as DP-SGD, struggle to handle relational data due to high sensitivity caused by entities participating in multiple relations and coupled sampling processes that complicate privacy amplification analyses.  To overcome these challenges, the authors propose a tailored DP-SGD variant that incorporates adaptive gradient clipping based on entity occurrence frequency and a privacy amplification analysis for a tractable subclass of coupled sampling. These innovations ensure rigorous privacy guarantees while maintaining strong utility in relational learning tasks. \n\nThe framework is validated through experiments on fine-tuning pre-trained text encoders using text-attributed relational datasets.  Results demonstrate that the proposed method achieves better privacy-utility trade-offs compared to standard DP-SGD approaches.  The paper also provides detailed sensitivity analyses, theoretical proofs, and privacy amplification bounds, ensuring the robustness of the proposed method.  Additionally, the authors explore the impact of key parameters, such as node degree capping and the number of negative samples, on model performance and privacy guarantees.  By advancing the theory and practice of privacy-preserving machine learning for graph data, this work contributes to safeguarding sensitive information while enabling meaningful relational inference in real-world applications.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Extensive theoretical analysis and experimental results are conducted on the proposed method, providing a thorough study of the method.\n- The proposed method provides an optimization process for relational learning under strong privacy protection of node-level DP.\n- The proposed mechanism can be easily adapted from an existing mechanism (DP-SGD).\n\n## Weaknesses\n\n- The analysis of local sensitivity is conducted on a scenario where any node $u^*$ sampled in an edge tuple $B$ must only exist in one edge tuple $T_i$ (based on lines 157-160). However, the proposed mechanism never discusses how to ensure this happens in Algorithms 1, 2, or 3. Furthermore, in this scenario, the sampling of the batch $B$ does not follow Poisson sampling anymore because it has to ensure any node $u$ only appears in one edge tuple $T_i$, leading to the uncertainty of the privacy guarantee of Alg. 3.\n- Although the proposed method leverages an adaptive clipping method, it is not very efficient. Specifically, in the clipping process of Alg. 2, after adding the $max-freq(T_i)$ to the clipping threshold, it seems like the new threshold is always smaller than $C$, incurring more information loss from the gradient. Furthermore, the noise added to the gradient is still $N(0, \\sigma^2C^2I)$, which is not scaled to the local sensitivity. Although the DP protection stays the same as the original DP-SGD, the information loss is higher due to the lower gradient clipping threshold.\n- The privacy application analysis in Theorem 4.1 ignores the impact of multiple training rounds $T$. This is a critical point because the training process is conducted with multiple training rounds, which incur more privacy amplification as discussed in DP-SGD.\n- The experiments do not consider other state-of-the-art baselines for both node-level DP (e.g., Sajadmanesh et al. [1]) and entity-level DP (e.g., Lai et al. [2]).\n- The presentation of the paper is hard to follow. The mathematical proofs of the theorems and propositions are very confusing, with lots of notation that was never introduced in the main body.\n\n[1] Sajadmanesh, Sina, et al. \"{GAP}: Differentially private graph neural networks with aggregation perturbation.\" 32nd USENIX Security Symposium (USENIX Security 23). 2023.\n\n[2] Lai, Phung, et al. \"User-entity differential privacy in learning natural language models.\" 2022 IEEE International Conference on Big Data (Big Data). IEEE, 2022.\n\nQUESTIONS:\nPlease focus on addressing the weaknesses raised above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-22"}