Model: kamkol/ab_testing_finetuned_2_arctic_ft-711c8143-04b7-4ecb-8e01-8847274c1d9c
Author: kamkol
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:250, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:Snowflake/snowflake-arctic-embed-l, base_model:finetune:Snowflake/snowflake-arctic-embed-l, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: Snowflake/snowflake-arctic-embed-l
Card metadata: tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:250, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: Snowflake/snowflake-arctic-embed-l
widget: source_sentence: How does erasing or losing cookies affect a user's User ID and experiment variant assignments?
sentences: who appear in that period are a realistic mix of consistent 
users who had 10 weeks to adapt to the no -underline 
experience, and new users who recently joined. Having this 
mix is important because consistent and new users may react 
to the change very differently. Consistent u sers are used to 
the old experience needing time to adapt, while new users do 
not have such primacy effects as they were not exposed to the 
old experience. Including new users is also important because 
some of these users are actually old users who churned their 
cookies (see Section III), and therefore are subject to learning 
in the same way as consistent users., Appears in IEEE Big Data.  Paper available at http://bit.ly/expLongTerm  
 
For more discussion on reasons for cookie churn and 
fingerprinting, see the treatment in Host Fingerprinting and 
Tracking on the Web: Privacy and Security Implication [23]. 
Cookie deletion rates are hard to estimate.  Using a panel 
of 400,000 home PCs, comScore estimated that 31 percent of 
U.S. Internet users cleared their first -party cookies during a 
month [24].   Anirban Dasgupta etal. [25] showed similar 
levels of cookie clearing (25% -33% monthly, depending on, purchase in the session, and thus depend on stable cookies. 
Users who erase or lose their cookies will lose their User 
ID and get a new one assigned on their next browser request, 
resulting in new experiment variant assignments. “Private 
browsing” (called InPrivate in Internet Explorer and Edge, 
Incognito in Chrome, and Private in Firefox) results in a 
short-lived identity: every such session start s with no 
historical cookies and thus has a unique identity until the 
private browser window is closed.   Another way in which 
users erase their cookies is by selecting the browser option to 
“Delete browsing history on exit” (Internet Explorer and, source_sentence: What challenges does survivorship bias present when analyzing long-term experiments and measuring user learning?
sentences: Because HTTP is a stateless protocol, cookies are used to 
maintain state.  In particular, many sites create a User ID 
cookie, which persists across browser requests and sessions. 
This cookie is then used to assign users to experiments so that 
their experience remains consistent for the duration of the 
experiment, i.e., there is a mapping from each cookie to the 
experiment(s) variants a user is in. 
Sites that require users to authenticate at the beginning of 
a session, such as e-mail sites and banks, have a known and 
stable identity.  However, many sites, including Google and, survivorship bias is a key issue in analyzing long -term 
experiments which, if present, makes it very hard  to 
accurately measure user learning.   
3. Deep analysis of a long -running experiment that we 
believe is impacted by “ user learning,” including a 
comparison of several approaches to  estimating the 
overall long -term impact.  We discuss the issue s of 
selection bias and perceived trends in the context of this 
experiment. 
4. Discussion of the impact of side effects and seasonalit y 
on the results of long -term experiments, using  actual 
experiments from Bing and MSN., Drilling by browser, we saw that the degradations were 
heavily concentrated in IE8 and IE9 as shown in Fig. 1. For 
Sessions/user, the differences were extremely statistically 
significant with p -value smaller than  10−10 , whereas for 
Chrome, Firefox, IE6, IE7, and Safari, they were not 
statistically significant. 
Due to the way IE handles permanent cookies with a 
memory mapped file, updating cookies increases the chance 
that a version mismatch between a domain cookie file and the 
index in memory will occur, and that all cookies in a domain, source_sentence: What factors, besides "user learning," contribute to the delta between the last and first data points on the chart?
sentences: week periods.  We note that the delta between the last and the 
first data point on the chart is due to not only “user learning” 
but also other factors such as seasonality, system changes, 
and the presence of users who did not have an opportunity to 
learn. As the FLT paper notes, this makes it hard to estimate 
the amount of impact due to “user learning” in this setup.  
Some metrics, such as Ad CTR, seem to show a trend that 
may reflect user learning; other metrics, however, do not 
show trends. One should be careful when trying to infer, ESTIMATING OVERALL LONG-TERM IMPACT 
 
FLT Our Approach 
Estimate Estimate 95% C.I. Bound 
Overall CTR 0.02% -0.17% (-0.21%, -0.13) 
Overall Clicks / 
User 
Not stat 
sig 
Not stat 
sig 
(-0.04%, 0.32%) 
Ad CTR -0.99% -1.47% (-1.65%, -1.29%) 
Ad Clicks / User -0.04% -1.45% (-1.67%, -1.23%), experiments to evaluate the impact of removing underlines 
from all links on Bing’s search results page. Fig. 3 shows the 
Treatment without underlines, Fig. 4 shows the Control with 
underlines. 
All our key metrics in short -term experiments showed 
that underline removal is not good for users: clickthr ough 
rates were down, time to a long dwell-time click degraded, 
users scrolled more, and monetization suffered [31]. This is 
even after we implemented features to show underlines on 
hover, which helped improve the key metrics a bit., source_sentence: What is the impact of side effects on the assumption that control and treatment groups are treated the same way after an experiment ends?
sentences: of side effects, discussed below. 
VIII. SIDE EFFECTS 
All discussion up to this point assumed that , once the 
experiment ends, the system will treat control and treatment 
groups in the same way. This allows attributing changes in 
the post-period to user learning. The assumption, however, is 
not correct when the treatment has a side effect that updates 
some information about the user, which stays after the 
experiment ends.  
Imagine an experiment testing a feature that warns people 
about traffic home.  Most users do  not have their home 
address in the system, so the treatment prompts for that., showed that these overly narrow estimates led scientists to 
interpret deviations over time as trends. In 1935, Edmonson 
proposed that the speed of light varied sinusoidally with a 40-
year perio d. In 1942, Birge widened then -recent estimates 
and provided a ‘steady state’ estimate.  But just nine years 
later, newer measurements were 2.4 standard deviations 
higher than Birge’s value, leading Rush in 1955 to claim that 
the speed of light was increasing.  Over time, measurements 
improved significantly and today we believe that the velocity, 7. Kohavi, R. , et. al . "Online Experimentation at Microsoft. " Third 
Workshop on Data Mining Case Studies and Practice Prize. 2009.  
8. Amatriain, X. and Basilico, J. "Netflix Recommendations: Beyond the 5 
stars." 2012. http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-
5-stars.html. 
9. McFarland, C. "Experiment!: Website conversion rate optimization with 
A/B and multivariate testing." s.l. : New Riders, 2012. 978-0321834607., source_sentence: What are the estimated monthly rates of first-party cookie deletion among U.S. Internet users according to comScore and Anirban Dasgupta et al.?
sentences: Tracking on the Web: Privacy and Security Implication [23]. 
Cookie deletion rates are hard to estimate.  Using a panel 
of 400,000 home PCs, comScore estimated that 31 percent of 
U.S. Internet users cleared their first -party cookies during a 
month [24].   Anirban Dasgupta etal. [25] showed similar 
levels of cookie clearing (25% -33% monthly, depending on 
geography) based on the Yahoo toolbar.  Such rates imply 
that long studies (e.g., months), where users are identified 
based on cookies could have a selection bi as problem – as, both directions: when we show fewer ads, users increase 
engagement, and when we show more ads, users decrease 
engagement or abandon. If users abandon at different rates 
between Control and Treatment, the remaining surviving 
population is different, and the conclusions can be 
completely wrong.  For example, the users generating the 
most revenue may get annoyed with more ads and abandon, 
leaving a surviving population with lower Revenue/user. 
3. Not taking into account selection bias [34].  
 In an online longitudinal study that relies on cookies for 
identification, there is likely to be a significant attrition due, statistically significant changes in metrics are observed 
during the post -period, they are assumed to be due the 
exposure to the treatment, and are called the “learning effect.” 
The authors note that due t o a number of factors (e.g. 
results being diluted due to cookies being an imperfect 
approximation of users) the methodology such as the one 
above underestimates the learning effect. They propose to use 
a “fudge factor” to compensate. The FLT paper notes that “in 
practice, we often use values of [fudge factor] between 2 and 
3 for desktop and laptop devices.”
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-l
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: Unknown
type: unknown
metrics: type: cosine_accuracy@1
value: 0.5
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.7419354838709677
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.7903225806451613
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.8629032258064516
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.5
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.2473118279569892
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.15806451612903222
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.08629032258064516
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.5
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.7419354838709677
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.7903225806451613
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.8629032258064516
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.689455738685852
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.633387736815156
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.6396895266032556
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 8
Created: 2025-05-11 18:36:23