Curriculum Disentangled Recommendation with Noisy Multi-feedback

1 Introduction . Recommenders aim to capture the user ’ s preferences from different aspects of information for more accurate prediction [ 1–4 ] . Learning disentangled representations that can uncover and disentangle the latent explanatory factors hidden in user behavioral data has recently been shown as an effective way to discover users ’ intentions , improving both recommendation accuracy and explainability [ 5–9 ] . Multi-feedback , normally including the positive feedback ( e.g. , click ) and negative feedback ( unclick and dislike , etc . ) , is of great significance to depict the user ’ s unbiased and various intentions [ 10 ] . Learning disentangled representation from multi-feedback is able to capture various user intentions more accurately , leading to improvement of accuracy and explainability in recommendation . However , learning such disentangled representation from multi-feedback that can best serve recommendation is quite challenging due to two reasons . i ) Multi-feedback is complex : different types of ∗Equal contributions †Corresponding Authors . 3Our code will be released at https : //github.com/forchchch/CDR 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feedback in multi-feedback data have complex relations with each other , and the relations between multi-feedback and various user intentions are also complicated . For instance , a user may unclick an item because of disinclination , getting tired of seeing the same type of item previously clicked too many times , as well as lacking enough time to view the item in detail . ii ) Multi-feedback is noisy : the large amount of feedback such as unclick brings a lot of noisy information ( useless for recommendation ) that may severely deteriorate the recommendation accuracy . For example , a user may unclick an item because he truly dislikes this item or he has interests in the item but is interrupted by others before making the click decision . As such , completely incorporating the unclick feedback as historical features to extract user intention representations may run the risk of bringing feature-level noise , while directly regarding the unclick behavior as negative samples to train the model will probably introduce label-level noise . Existing works on disentangled representation learning merely rely on the positive user feedback to extract the user intentions , failing to handle the complex relations and noisy ( useless ) information hidden in multi-feedback data . To tackle these challenges , we propose a Curriculum Disentangled Recommendation ( CDR ) model that is able to accurately discover various user intentions from different kinds of user feedback . The CDR model consists of two core components , i.e. , the co-filtering dynamic routing mechanism and the adjustable self-evaluating curriculum strategy , which together learn disentangled representations as well as captures the complex relational dependencies and filters out noise from multi-feedback to achieve more accurate recommendation . More concretely , the proposed routing mechanism utilizes the dependencies among different feedback to accurately discover the user preferences , followed by the intention aggregation which further helps to learn disentangled representations for users . Our adjustable self-evaluating curriculum then guides the model towards better optima by reweighing samples according to the self-evaluated difficulty . The proposed curriculum can further enhance the model performance by making the model learn from multi-feedback data of different difficulties at different learning paces , thus preventing the model from overfitting the data with improper labels . We conduct extensive experiments on several real-world datasets to demonstrate that our proposed CDR model can significantly beat baseline methods in terms of recommendation accuracy . To summarize , our contributions are listed as follows . ( 1 ) We propose a Curriculum Disentangled Recommendation ( CDR ) model to learn disentangled representations for recommendation from multi-feedback with complex relations and noisy information . ( 2 ) We propose a co-filtering dynamic routing mechanism to simultaneously capture complex relations among different sorts of feedback and various user intentions as well as filtering out feature-level noise in multi-feedback . ( 3 ) We propose an adjustable self-evaluating curriculum to guide the model towards better optima by alleviating the impact of label-level noise in a more controllable and convenient way . 2 Related Work . Recommendation Based on Multi-Feedback Apart from positive feedback , negative feedback data are also essential for capturing user preference [ 11 ] . Early efforts [ 12–14 ] regard all unclick data equally as negative feedback and decrease the confidence compared to click data . Later works utilize additional information [ 15–18 ] or reinforcement learning [ 19–21 ] to distinguish real negative signals from all the unclick data . A recent method [ 22 ] also recognizes negative feedback from the clicked news by reading dwell time . However , all these methods merely select better negative training samples and ignore multi-feedback as features to learn user interests . DFN [ 10 ] simultaneously incorporate click , unclick , and dislike feedback simply by rough attention mechanism and concatenation , which is insufficient to capture the user ’ s comprehensive and accurate interests . Disentangled Representation Learning Disentangled representation learning aims to learn various hidden explanatory factors behind observable data in different parts of the learned vector presentation [ 23 ] . Many variants of variational auto-encoder ( VAE ) [ 24 ] have been studied to improve the disentanglement of the learned representation [ 25–27 ] by adding regularization terms to decrease the mutual information between different parts of the learned vector . Disentangled representation learning has also found its application in recommendation [ 5–8 , 28 , 29 ] by learning disentangled user preference from user positive feedback to improve both the performance and interpretability . Different from these works , our work focuses on learning more comprehensive disentangled interests with multi-feedback . Curriculum Learning Curriculum learning [ 30 , 31 ] aims to design a dynamic sample reweighting strategy throughout the training process to improve performance and training efficiency . Traditional methods mostly follow an easy-to-hard paradigm , i.e. , assigning higher weights to easier samples in earlier training , where the “ easiness ” measurement can be both domain-knowledge-based [ 32– 34 ] and loss-based [ 35 , 36 ] . These methods can only be effective for specific tasks , and “ easyto-hard ” is sometimes sub-optimal compared with “ harder first ” [ 37 ] . Recent works propose to automatically learn a curriculum by reinforcement learning [ 38–40 ] , meta-learning [ 41 ] , etc . However , the optimization process of these curricula is time- or resource-consuming , which is too costly for abundant recommendation data . Our method provides an efficient and also flexible curriculum that is beyond the easy-to-hard limit and more adjustable and interpretable than automatic methods . 3 Method . In this section , we introduce the proposed Curriculum Disentangled Recommendation ( CDR ) model ( Figure 1 ) to mine comprehensive and accurate user intentions from users ’ noisy multi-feedback . 3.1 Notations and Problem Formulation . Notations We denote φ ( a , b ) as the inner product of two vectors LayerNorm ( a ) and LayerNorm ( b ) , where LayerNorm refers to Layer Normalization operation and φ ( a , b ) measures the similarity between vector a and b. sim ( keyi , query ) is the normalized similarity between query and keyi on set Q that is composed of all the keys : sim ( keyi , query ) = exp ( φ ( keyi , query ) ) ∑ i′∈Q exp ( φ ( keyi′ , query ) ) . Multi-feedback based prediction The vth user ’ s historical behavior contains his or her clicked item sequence c ( v ) = [ c ( v ) 1 , c ( v ) 2 , · · · , c ( v ) m ] , unclicked ( i.e. , presented but not clicked ) item sequence u ( v ) = [ u ( v ) 1 , u ( v ) 2 , · · · , u ( v ) n ] , and disliked ( i.e. , press to the dislike button or low rated ) item sequence d ( v ) = [ d ( v ) 1 , d ( v ) 2 , · · · , d ( v ) l ] , where each term in these sequences represents an item that the v ( th ) user interacted with . Our goal is to learn users ’ disentangled intentions and then accurately predict svt , i.e. , the vth user ’ s preference towards the candidate item t ( v ) . Besides , we utilize the profile ( e.g. , age and gender ) of the vth user to aid the learning and prediction process . 3.2 Co-filtering Dynamic Routing . The routing mechanism takes as input user profile , candidate item feature , and user multi-feedback history , and makes the final prediction by the following three steps . It first utilizes the relations behind different kinds of feedback to discover where the user ’ s true interests locate . Then , the model aggregates the user ’ s disentangled intentions from the useful behavior . Finally , it predicts the user ’ s preference towards the candidate item based on the learned intentions . 3.2.1 Interests Mining . The user ’ s interests could be reflected by his or her various kinds of feedback . Ignoring any kind of feedback may lead to incomplete or inaccurate preference modeling . However , the noise hidden behind these data makes it infeasible to directly use multi-feedback to learn the user ’ s interests . Thus , how to make use of the relations of different kinds of feedback to discover where the user ’ s true interests locate is the key in this step . First , we project each item in c ( v ) , d ( v ) , and u ( v ) into the embedding space by concatenating its ID and category embedding . Thus , we obtain the clicked embedding sequence h ( v ) c = [ h ( v ) c1 , h ( v ) c2 , · · · , h ( v ) cm ] , unclicked embedding sequence h ( v ) u = [ h ( v ) u1 , h ( v ) u2 , · · · , h ( v ) un ] , and disliked embedding sequence h ( v ) d = [ h ( v ) d1 , h ( v ) d2 , · · · , h ( v ) dl ] . We also project the information of the user profile into the embedding space and obtain the user profile feature F ( v ) = { F ( v ) 1 , F ( v ) 2 , · · · , F ( v ) g } , where g is the number of user profiles . We then utilize the power of transformer encoder [ 42 ] to obtain better representation for each item with fully interactions : [ z ( v ) c1 , z ( v ) c2 , · · · , z ( v ) cm ] = C-Encoder ( h ( v ) c ) , [ z ( v ) u1 , z ( v ) u2 , · · · , z ( v ) un ] = U -Encoder ( h ( v ) u ) , [ z ( v ) d1 , z ( v ) d2 , · · · , z ( v ) dl ] = D-Encoder ( h ( v ) d ) , where the z ( v ) ci , z ( v ) ui , z ( v ) di refer to ddimensional feature vectors containing the vth user ’ s interests . We then need deeper thinking towards the relations among multiple feedback to discover user ’ s true interests . The first thing to consider is where the user ’ s interests will locate . We note that not only the click behavior contains user interests , but the unclick sequence generated by the recommender algorithms could also contain rich information about user interests . Therefore , utilizing both kinds of feedback could help to learn comprehensive user interests . The second noteworthy thing is how to discover the user ’ s true interests from the noisy data . To filter out the noise information in these historical sequences , we need the disliked item sequences that could reflect the user ’ s strong negative intentions and have high confidence . Thus , we average these strong negative features and regard it as the negative intention of the vth user : n ( v ) = 1l ∑l i=1 z ( v ) di . This highly confident negative intention could be used to filter the noise in both clicked and unclicked history items . We adopt the similarity between each item and the negative intention to judge whether the item should contribute to the user intentions . Based on the intuition that higher similarity to the negative intention should contribute less to the user preference , the contribution of each item to the user intentions can be formulated as follows : n ′ ( v ) c = n ( v ) + bc1 , z ′ ( v ) ci = z ( v ) ci + σ1 ( Wc1z ( v ) ci + bc2 ) , d ( v ) ci = exp ( −φ ( z ′ ( v ) ci , n ′ ( v ) c ) ) ∑m j=1 exp ( −φ ( z ′ ( v ) cj , n ′ ( v ) c ) ) , n ′ ( v ) u = n ( v ) + bu1 , z ′ ( v ) uj = z ( v ) uj + σ1 ( Wu1z ( v ) uj + bu2 ) , d ( v ) uj = σ ( −φ ( z ′ ( v ) uj , n ′ ( v ) u ) +MLP ( [ F ( v ) ; z ( v ) uj ] ) ) , ( 1 ) where σ1 is the ReLU activation function and σ is the Sigmoid activation function . d ( v ) ci and d ( v ) uj represent the contribution of each clicked item and unclicked item to the user intention , respectively . We obtain d ( v ) ci by calculating the similarity between each clicked item and the negative intention , and we also introduce the parameters Wc1 , bc1 , bc2 to avoid some outliers that disobey our intuition . As for the unclicked item sequence , there exists more noise . Thus , we utilize not only the information of the user negative intention but also the user profile feature F ( v ) to measure the importance of each unclicked item . We first concatenate user profile features and the unclicked item feature together , and then use a two-layer MLP to judge whether the unclicked item is important . Besides the noise effect , we also take the time and candidate item factors into consideration . Intuitively , more recently clicked items and items having a higher similarity to the candidate item will contribute more to the user preference towards the candidate item . The formulation is as follows : time ( v ) ci = z ( v ) ci + σ1 ( Wc2 [ z ( v ) ci ; pi ] + bc3 ) , time ′ ( v ) cm = z ( v ) cm + σ1 ( Wc3 [ z ( v ) cm ; pm ] + bc4 ) , cand ( v ) ci = z ( v ) ci + σ1 ( Wc4z ( v ) ci + bc5 ) , z ′ ( v ) ct = h ( v ) t + bc6 , f ( v ) ci = sim ( time ( v ) ci , time ′ ( v ) cm ) + sim ( cand ( v ) ci , z ′ ( v ) ct ) , cand ( v ) uj = z ( v ) uj + σ1 ( Wu2z ( v ) uj + bu3 ) , z ′ ( v ) ut = h ( v ) t + bu4 , f ( v ) uj = sim ( cand ( v ) uj , z ′ ( v ) ut ) . ( 2 ) We consider the impact of both time and candidate factors for the clicked items . The candidate item feature h ( v ) t ∈ Rd and the most recently clicked item feature z ( v ) cm measures the importance of each clicked item . pi is the position embedding of each item and encoded for more accurate time factor consideration . However , for the unclick sequence , we only consider the candidate factor , because the time of unclicking one item has low relations with the user ’ s current interests . f ( v ) ci and f ( v ) uj are the importance of each clicked item and unclicked item , respectively .

Summary: This paper proposes a model to learn disentangled representations for user intentions from complex and noisy multi-feedback. The model can capture complex relations among multi-feedback and various user intentions, and filter out feature-level noise in multi-feedback. Moreover, this paper proposes an adjustable curriculum to alleviate the impact of label-level noise in a more controllable way.