Project: 047_More_RLHF_More_Trust_On_The_Impact_of_Preference_A | Function: DPO_loss (iclr)

system: You are an expert Python programmer implementing algorithms from scientific papers.
Complete the masked function based on the algorithm description.
Follow Python best practices and maintain consistent code style.
Return **only** the complete Python function code.

user: Complete this Python function:

**DESCRIPTION:** Implements Direct Preference Optimization (DPO) loss function, a core RLHF algorithm that optimizes language model policy directly using preference data without explicit reward modeling

**KEY TERMS:** {
  "pi_theta": "πθ (current model)",
  "pi_sft": "πSFT (supervised fine-tuned model)",
  "y_w": "yw (preferred output sequence)",
  "y_l": "yl (non-preferred output sequence)",
  "beta": "β (scaling factor)"
}

**CONTEXT CODE:** import torch
import torch.nn as nn
import numpy as np
import os, pickle
from transformers import GPTNeoXForCausalLM, AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSequenceClassification
import pandas as pd
from datasets import load_dataset
from huggingface_hub import snapshot_download
import json
from tqdm import tqdm
from peft import PeftModel, PeftConfig
from safetensors.torch import load_file
from collections import defaultdict
from time import time
from torch.utils.data import DataLoader
import torch.nn.functional as F

device = "cuda:0" if torch.cuda.is_available() else "cpu"

def DPO_loss(pi_theta, pi_sft, x, y_w, y_l, beta=1.0):
    """
    Compute the DPO loss.

    Args:
    - pi_theta: The current model (parameterized by theta).
    - pi_sft: The supervised fine-tuned model (SFT).
    - x: The context input sequence.
    - y_w: The preferred output sequence.
    - y_l: The non-preferred output sequence.
    - beta: The scaling factor for the log-ratio term (default: 1.0).

    Returns:
    - loss: The DPO loss.
    """
    raise NotImplementedError("Function implementation has been masked for testing")

def load_pickle_safely(filename):
    data = []
    with open(filename, 'rb') as f:
        while True:
            try:
                data.append(pickle.load(f))
            except EOFError:
                break
            except pickle.UnpicklingError:
                print("Unpickling error encountered.")
                break
    return data

def normalize_infl(arr):
    # Copy the array to avoid modifying the original
    normalized_arr = np.zeros_like(arr, dtype=np.float64)

    # Positive values normalization (0, 1)
    pos_mask = arr > 0
    if np.any(pos_mask):
        pos_vals = arr[pos_mask]
        normalized_arr[pos_mask] = pos_vals / np.max(pos_vals)  # Scale by the maximum positive value

    # Negative values normalization (-1, 0)
    neg_mask = arr < 0
    if np.any(neg_mask):
        neg_vals = arr[neg_mask]
        normalized_arr[neg_mask] = neg_vals / -np.min(neg_vals)  # Scale by the minimum (most negative) value

    return normalized_arr

def influence_function_attribution(tr_grad_dict, val_grad_dict, lambda_const_param=10):
    """
    Implements efficient influence function-based data attribution method adapted for RLHF setting to analyze
    how training data impacts trustworthiness benchmarks.

    Args:
        tr_grad_dict: Dictionary containing training gradients
        val_grad_dict: Dictionary containing validation gradients
        lambda_const_param: Parameter for layer-wise lambda calculation (default: 10)

    Returns:
        if_tmp_dict: Dictionary containing influence scores for each training sample
    """
    raise NotImplementedError("Function implementation has been masked for testing")

Complete the function implementation replacing "raise NotImplementedError".