Curriculum Disentangled Recommendation with Noisy Multi-feedback

1 Introduction . Recommenders aim to capture the user ’ s preferences from different aspects of information for more accurate prediction [ 1–4 ] . Learning disentangled representations that can uncover and disentangle the latent explanatory factors hidden in user behavioral data has recently been shown as an effective way to discover users ’ intentions , improving both recommendation accuracy and explainability [ 5–9 ] . Multi-feedback , normally including the positive feedback ( e.g. , click ) and negative feedback ( unclick and dislike , etc . ) , is of great significance to depict the user ’ s unbiased and various intentions [ 10 ] . Learning disentangled representation from multi-feedback is able to capture various user intentions more accurately , leading to improvement of accuracy and explainability in recommendation . However , learning such disentangled representation from multi-feedback that can best serve recommendation is quite challenging due to two reasons . i ) Multi-feedback is complex : different types of ∗Equal contributions †Corresponding Authors . 3Our code will be released at https : //github.com/forchchch/CDR 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feedback in multi-feedback data have complex relations with each other , and the relations between multi-feedback and various user intentions are also complicated . For instance , a user may unclick an item because of disinclination , getting tired of seeing the same type of item previously clicked too many times , as well as lacking enough time to view the item in detail . ii ) Multi-feedback is noisy : the large amount of feedback such as unclick brings a lot of noisy information ( useless for recommendation ) that may severely deteriorate the recommendation accuracy . For example , a user may unclick an item because he truly dislikes this item or he has interests in the item but is interrupted by others before making the click decision . As such , completely incorporating the unclick feedback as historical features to extract user intention representations may run the risk of bringing feature-level noise , while directly regarding the unclick behavior as negative samples to train the model will probably introduce label-level noise . Existing works on disentangled representation learning merely rely on the positive user feedback to extract the user intentions , failing to handle the complex relations and noisy ( useless ) information hidden in multi-feedback data . To tackle these challenges , we propose a Curriculum Disentangled Recommendation ( CDR ) model that is able to accurately discover various user intentions from different kinds of user feedback . The CDR model consists of two core components , i.e. , the co-filtering dynamic routing mechanism and the adjustable self-evaluating curriculum strategy , which together learn disentangled representations as well as captures the complex relational dependencies and filters out noise from multi-feedback to achieve more accurate recommendation . More concretely , the proposed routing mechanism utilizes the dependencies among different feedback to accurately discover the user preferences , followed by the intention aggregation which further helps to learn disentangled representations for users . Our adjustable self-evaluating curriculum then guides the model towards better optima by reweighing samples according to the self-evaluated difficulty . The proposed curriculum can further enhance the model performance by making the model learn from multi-feedback data of different difficulties at different learning paces , thus preventing the model from overfitting the data with improper labels . We conduct extensive experiments on several real-world datasets to demonstrate that our proposed CDR model can significantly beat baseline methods in terms of recommendation accuracy . To summarize , our contributions are listed as follows . ( 1 ) We propose a Curriculum Disentangled Recommendation ( CDR ) model to learn disentangled representations for recommendation from multi-feedback with complex relations and noisy information . ( 2 ) We propose a co-filtering dynamic routing mechanism to simultaneously capture complex relations among different sorts of feedback and various user intentions as well as filtering out feature-level noise in multi-feedback . ( 3 ) We propose an adjustable self-evaluating curriculum to guide the model towards better optima by alleviating the impact of label-level noise in a more controllable and convenient way . 2 Related Work . Recommendation Based on Multi-Feedback Apart from positive feedback , negative feedback data are also essential for capturing user preference [ 11 ] . Early efforts [ 12–14 ] regard all unclick data equally as negative feedback and decrease the confidence compared to click data . Later works utilize additional information [ 15–18 ] or reinforcement learning [ 19–21 ] to distinguish real negative signals from all the unclick data . A recent method [ 22 ] also recognizes negative feedback from the clicked news by reading dwell time . However , all these methods merely select better negative training samples and ignore multi-feedback as features to learn user interests . DFN [ 10 ] simultaneously incorporate click , unclick , and dislike feedback simply by rough attention mechanism and concatenation , which is insufficient to capture the user ’ s comprehensive and accurate interests . Disentangled Representation Learning Disentangled representation learning aims to learn various hidden explanatory factors behind observable data in different parts of the learned vector presentation [ 23 ] . Many variants of variational auto-encoder ( VAE ) [ 24 ] have been studied to improve the disentanglement of the learned representation [ 25–27 ] by adding regularization terms to decrease the mutual information between different parts of the learned vector . Disentangled representation learning has also found its application in recommendation [ 5–8 , 28 , 29 ] by learning disentangled user preference from user positive feedback to improve both the performance and interpretability . Different from these works , our work focuses on learning more comprehensive disentangled interests with multi-feedback . Curriculum Learning Curriculum learning [ 30 , 31 ] aims to design a dynamic sample reweighting strategy throughout the training process to improve performance and training efficiency . Traditional methods mostly follow an easy-to-hard paradigm , i.e. , assigning higher weights to easier samples in earlier training , where the “ easiness ” measurement can be both domain-knowledge-based [ 32– 34 ] and loss-based [ 35 , 36 ] . These methods can only be effective for specific tasks , and “ easyto-hard ” is sometimes sub-optimal compared with “ harder first ” [ 37 ] . Recent works propose to automatically learn a curriculum by reinforcement learning [ 38–40 ] , meta-learning [ 41 ] , etc . However , the optimization process of these curricula is time- or resource-consuming , which is too costly for abundant recommendation data . Our method provides an efficient and also flexible curriculum that is beyond the easy-to-hard limit and more adjustable and interpretable than automatic methods . 3 Method . In this section , we introduce the proposed Curriculum Disentangled Recommendation ( CDR ) model ( Figure 1 ) to mine comprehensive and accurate user intentions from users ’ noisy multi-feedback . 3.1 Notations and Problem Formulation . Notations We denote φ ( a , b ) as the inner product of two vectors LayerNorm ( a ) and LayerNorm ( b ) , where LayerNorm refers to Layer Normalization operation and φ ( a , b ) measures the similarity between vector a and b. sim ( keyi , query ) is the normalized similarity between query and keyi on set Q that is composed of all the keys : sim ( keyi , query ) = exp ( φ ( keyi , query ) ) ∑ i′∈Q exp ( φ ( keyi′ , query ) ) . Multi-feedback based prediction The vth user ’ s historical behavior contains his or her clicked item sequence c ( v ) = [ c ( v ) 1 , c ( v ) 2 , · · · , c ( v ) m ] , unclicked ( i.e. , presented but not clicked ) item sequence u ( v ) = [ u ( v ) 1 , u ( v ) 2 , · · · , u ( v ) n ] , and disliked ( i.e. , press to the dislike button or low rated ) item sequence d ( v ) = [ d ( v ) 1 , d ( v ) 2 , · · · , d ( v ) l ] , where each term in these sequences represents an item that the v ( th ) user interacted with . Our goal is to learn users ’ disentangled intentions and then accurately predict svt , i.e. , the vth user ’ s preference towards the candidate item t ( v ) . Besides , we utilize the profile ( e.g. , age and gender ) of the vth user to aid the learning and prediction process . 3.2 Co-filtering Dynamic Routing . The routing mechanism takes as input user profile , candidate item feature , and user multi-feedback history , and makes the final prediction by the following three steps . It first utilizes the relations behind different kinds of feedback to discover where the user ’ s true interests locate . Then , the model aggregates the user ’ s disentangled intentions from the useful behavior . Finally , it predicts the user ’ s preference towards the candidate item based on the learned intentions . 3.2.1 Interests Mining . The user ’ s interests could be reflected by his or her various kinds of feedback . Ignoring any kind of feedback may lead to incomplete or inaccurate preference modeling . However , the noise hidden behind these data makes it infeasible to directly use multi-feedback to learn the user ’ s interests . Thus , how to make use of the relations of different kinds of feedback to discover where the user ’ s true interests locate is the key in this step . First , we project each item in c ( v ) , d ( v ) , and u ( v ) into the embedding space by concatenating its ID and category embedding . Thus , we obtain the clicked embedding sequence h ( v ) c = [ h ( v ) c1 , h ( v ) c2 , · · · , h ( v ) cm ] , unclicked embedding sequence h ( v ) u = [ h ( v ) u1 , h ( v ) u2 , · · · , h ( v ) un ] , and disliked embedding sequence h ( v ) d = [ h ( v ) d1 , h ( v ) d2 , · · · , h ( v ) dl ] . We also project the information of the user profile into the embedding space and obtain the user profile feature F ( v ) = { F ( v ) 1 , F ( v ) 2 , · · · , F ( v ) g } , where g is the number of user profiles . We then utilize the power of transformer encoder [ 42 ] to obtain better representation for each item with fully interactions : [ z ( v ) c1 , z ( v ) c2 , · · · , z ( v ) cm ] = C-Encoder ( h ( v ) c ) , [ z ( v ) u1 , z ( v ) u2 , · · · , z ( v ) un ] = U -Encoder ( h ( v ) u ) , [ z ( v ) d1 , z ( v ) d2 , · · · , z ( v ) dl ] = D-Encoder ( h ( v ) d ) , where the z ( v ) ci , z ( v ) ui , z ( v ) di refer to ddimensional feature vectors containing the vth user ’ s interests . We then need deeper thinking towards the relations among multiple feedback to discover user ’ s true interests . The first thing to consider is where the user ’ s interests will locate . We note that not only the click behavior contains user interests , but the unclick sequence generated by the recommender algorithms could also contain rich information about user interests . Therefore , utilizing both kinds of feedback could help to learn comprehensive user interests . The second noteworthy thing is how to discover the user ’ s true interests from the noisy data . To filter out the noise information in these historical sequences , we need the disliked item sequences that could reflect the user ’ s strong negative intentions and have high confidence . Thus , we average these strong negative features and regard it as the negative intention of the vth user : n ( v ) = 1l ∑l i=1 z ( v ) di . This highly confident negative intention could be used to filter the noise in both clicked and unclicked history items . We adopt the similarity between each item and the negative intention to judge whether the item should contribute to the user intentions . Based on the intuition that higher similarity to the negative intention should contribute less to the user preference , the contribution of each item to the user intentions can be formulated as follows : n ′ ( v ) c = n ( v ) + bc1 , z ′ ( v ) ci = z ( v ) ci + σ1 ( Wc1z ( v ) ci + bc2 ) , d ( v ) ci = exp ( −φ ( z ′ ( v ) ci , n ′ ( v ) c ) ) ∑m j=1 exp ( −φ ( z ′ ( v ) cj , n ′ ( v ) c ) ) , n ′ ( v ) u = n ( v ) + bu1 , z ′ ( v ) uj = z ( v ) uj + σ1 ( Wu1z ( v ) uj + bu2 ) , d ( v ) uj = σ ( −φ ( z ′ ( v ) uj , n ′ ( v ) u ) +MLP ( [ F ( v ) ; z ( v ) uj ] ) ) , ( 1 ) where σ1 is the ReLU activation function and σ is the Sigmoid activation function . d ( v ) ci and d ( v ) uj represent the contribution of each clicked item and unclicked item to the user intention , respectively . We obtain d ( v ) ci by calculating the similarity between each clicked item and the negative intention , and we also introduce the parameters Wc1 , bc1 , bc2 to avoid some outliers that disobey our intuition . As for the unclicked item sequence , there exists more noise . Thus , we utilize not only the information of the user negative intention but also the user profile feature F ( v ) to measure the importance of each unclicked item . We first concatenate user profile features and the unclicked item feature together , and then use a two-layer MLP to judge whether the unclicked item is important . Besides the noise effect , we also take the time and candidate item factors into consideration . Intuitively , more recently clicked items and items having a higher similarity to the candidate item will contribute more to the user preference towards the candidate item . The formulation is as follows : time ( v ) ci = z ( v ) ci + σ1 ( Wc2 [ z ( v ) ci ; pi ] + bc3 ) , time ′ ( v ) cm = z ( v ) cm + σ1 ( Wc3 [ z ( v ) cm ; pm ] + bc4 ) , cand ( v ) ci = z ( v ) ci + σ1 ( Wc4z ( v ) ci + bc5 ) , z ′ ( v ) ct = h ( v ) t + bc6 , f ( v ) ci = sim ( time ( v ) ci , time ′ ( v ) cm ) + sim ( cand ( v ) ci , z ′ ( v ) ct ) , cand ( v ) uj = z ( v ) uj + σ1 ( Wu2z ( v ) uj + bu3 ) , z ′ ( v ) ut = h ( v ) t + bu4 , f ( v ) uj = sim ( cand ( v ) uj , z ′ ( v ) ut ) . ( 2 ) We consider the impact of both time and candidate factors for the clicked items . The candidate item feature h ( v ) t ∈ Rd and the most recently clicked item feature z ( v ) cm measures the importance of each clicked item . pi is the position embedding of each item and encoded for more accurate time factor consideration . However , for the unclick sequence , we only consider the candidate factor , because the time of unclicking one item has low relations with the user ’ s current interests . f ( v ) ci and f ( v ) uj are the importance of each clicked item and unclicked item , respectively .

Summary: The paper introduces an approach to take into account different types of positive and negative feedback in a recommendation model, while accounting for the possible noise contained in the feedback. The proposed model relies in particular on three types of feedback (click, lack of click and dislike). The sequences of different types of feedback are first encoded with a Transformer and then aggregated through a series of MLPs and attention mechanisms into vectors supposed to capture and disentangle the intents of the user.  Independently from this module, the paper also proposes a curriculum learning approach to train the model and allocate with flexibility more weight on either the easier samples or the harder samples at the beginning of the training.  The proposed approach is evaluated on 3 public datasets and 1 proprietary dataset, and compared against several baselines.