Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yamien Cheng
authors: Jheng-Long Wu
reviews: 
Comments: The paper proposes ThreadsGAN, a model built with BERT, transformer, CNN and GAN, aiming for incorporating multi-topic and social response intention tags to tackle thread generation task. The paper claims that the proposed model has strong performance for generating realistic and appropriate discussion threads.
Strengths: 1. ThreadsGAN integrates GAN with topic and response tags to enhance discussion thread coherence and diversity. It is novel to use GAN as a text generation model. 
2. The paper collected dataset from PTT website, which is across multiple subjects and has a large size of threads. This data is useful to train and evaluate models on thread generation task.
Weaknesses: Overall this paper needs a major revision. I list my comments below:
1. Weak motivation. I am not convinced by the paper why thread generation is an important task to tackle with. The paper did not cite any related work in thread generation (Line 35), nor explained the usability or necessity for downstream application. How thread generation is different from conversation generation, or question answering? Why previous works cannot solve this task?
2. Lacks clarity in methodology. The paper did not formally explain the task - what is expected input and output. The authors explained the model's input in Section 3.2.1, but without a formal definition of the task it was hard to understand. For example, the paper mentioned 'year' the first time in 3.2.1, which wasn't mentioned before, and the role of year is not explained. 
3. Missing explanation of model design. The proposed model has a lot of components, like BERT, CNN and GAN. There are a lot of alternatives that are generally better than BERT (RoBERTA) and CNN (LSTM), but why using these models as modules are not explained nor compared. 
4. Weak performance: Both automatic and human evaluation showed that the proposed model is worse than GPT4, and even SeqGAN in some criteria. It is not explained why the proposed method better (if not in performance, then in cost, explanability...), or takeaways of using GAN as a generation model. Furthermore, the baselines are too weak to compare. The paper used unprompted, off-the-shelf GPT-4 (since the paper did not mention any adjustment), which I think will perform stronger with a well-designed prompting or SFT.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This paper proposes a GAN-based method to train a generative model that generates discussion threads. The model architecture is largely based on BERT but modifications were made to take into consideration the thread structure, explicitly separating the main post, the previous response, and the next response. A hidden representation of topics is also employed into the generation process. A discriminator model is trained concurrently to distinguish generated threads and real threads to help improve the generation model. Evaluation is done on data collected from a Taiwanese online forum. The proposed method is compared with SeqGAN and LLMs. The results are mixed. The proposed method doesn’t outperform GPT-4 but performs better than some other baselines based on some of the metrics.
Strengths: The paper proposes a new customized method for a special task.
Weaknesses: - The proposed method is not sound, based on my understanding.

Although the high-level idea of using GAN to develop a thread generation model is fine, the model details seem to have some issues. Specifically, the encoder takes in the next response as input (formula 3.2.3), but the decode generates the next response (formula 3.2.9), that is, the output is given as the input. Isn’t this a problem? I might have misunderstood the model, but the description of the model is not clear enough. 

- The presentation needs improvement.

As mentioned above, the description of the method is not clear enough to resolve the main question I have about the validity of the generation process. Other than the method section, the other sections also lack important details and are sometimes not clear. For example, how are LLMs used to generate the threads? Do you use a prompt and if so, how does the prompt look like? 
Most importantly, while the introduction section discusses several challenges for discussion thread generation, such as ensuring authenticity and coherence of the generated posts and ability to adapt to diverse topics, it is not clearly explained why the proposed method addresses these challenges.

- Lack of motivation.

The task studied is not well motivated. The proposed solution is not linked to the identified challenges and therefore is not well motivated, either.

- The work has limited contribution. 

First, it is not clear to me why discussion thread generation is important and what its applications are. Although the authors try to motivate the work in the introduction section, there are very limited references supporting the claims that discussion thread generation is important and that much work has been done on this problem. The few cited papers are also old and do not represent the state-of-the-art solutions to text generation problems. The proposed method itself has some issues as pointed out above in the soundness section. In addition, the results do not show advantage of the proposed method over LLMs, which are now widely adopted for many tasks. Therefore, it is hard to imagine what other researchers can learn from this work.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 2
reviews: 
Comments: The paper presents ThreadsGAN, a generative model designed to improve the quality of discussion threads by focusing on coherence, interactivity, and diversity. The model is evaluated against several baseline models, including SeqGAN, TAIDE, and GPT-4, using both traditional metrics and human evaluations.
Strengths: N/A
Weaknesses: The quality of the paper is very poor. The logical coherence between the paragraphs of the paper is poor and does not conform to the conventional writing style of a paper. The method proposed in the paper lacks any innovation. The experimental data in the paper also shows that the proposed method has poor performance.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This submission examines the generation of discourse discussions threads using a Generative Adversarial Network (GAN).  The authors condition on topic and social response tags to help guide the generation process.  The work grounds on a Sequence GAN model for discourse and applies it to a parenting forum found in Taiwanese social media, and validate using human evaluation.
Strengths: * The work examines an interesting application of discussion thread generation.
* The model uses relatively lightweight models for the characterisation (BERT) and generation, incorporating a CNN layer.
* The work performs a human evaluation of the model and baselines.
Weaknesses: * The related work section needs more contextualisation before getting into the specifics of individual works.  It is not clear of the organisational structure and overall relevances of which works were reviewed and why. How they are organised and presented is crucially missing -- e.g., the GAN (S2.2) section is a mismatch of single sentence summaries that do not present a coherent argument for understanding the prior work and how it motivates the current submission.  
* The provenance of the data (source platform and actual chosen topics) was not clear from (L140-161).  The quality guidelines and how interannotator agreement on the application of the guidelines was not elaborated on, so the veracity and quality of the data are suspect.
* There are design choices that are not well defended.  The authors decide to use BERT as part of their architecture, but choose to use a decoder-only transfomer architecture for generation, and the more natural choice of using a compatible embedding model that would be congruent with the decoder backend, was not defended.  I would need a more strong motivation and defense of the design choice. 
  * Similarly, the use of a CNN for thread generation is not defended.  The work is generating a sequence of tokens, so it's not clear why a patch oriented model (such as a CNN) is used.  
* The dataset is somewhat limited and exactly why the authors applied the work only to this particular domain and discussion board was not clear.  Fig. 1 further shows 4 of the 10 boards as having fewer than 10 viable posts, so it's not clear that the study's output generalises well.
* The descriptions of the baselines are insufficient to allow reproducibility.
* The ThreadGAN model does not perform well against commercial methods (GPT-4) or its basic baseline (SeqGAN), which would need to be demonstrated for publication.
* The human evaluation does not discuss inter-annotator agreement, which is a concern.  Without this aspect the quality of the evaluation is not assured, and the veracity and seriousness of the human evaluation is undermined.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1